Homework — Assignment 6   Inside Out Surveys and Questionnaires

Read the excerpts from the Resources. Find some other advice about designing and administering questionnaires in whatever format you like, text, video, interview an expert.

Deliverable(s)

Blog what you learned about designing and administering questionnaires. If you have personal experience, please feel free to share that with us as well. If you have some initial questionnaire questions for your final project, share them with us.

 

• A. What I have learned about surveys & designing and administering questionnaires.

 

1. What is a survey?

According to Observing the User Experience, a survey is a set of questions that allows a large group of people to describe themselves, their interests, and their preferences in a structured way. Using statistical tools on the results, UX designers can reveal broad characteristics about their users and extract interesting patterns. But, if UX designers do not design the survey carefully, they can ask the wrong people the wrong questions, producing results that are inaccurate, inconclusive, or even deceptive. Web-based surveys are especially vulnerable because, lacking any direct contact with respondents themselves, their accuracy relies on their ability and willingness to honestly report their opinions. Without direct contact (whether through in-person visits or analysis of behavioral data), UX designers cannot tell whether respondents’ descriptions of themselves or what kind of service they really use.

 

2. What is a questionnaire？

A questionnaire is a research tool consisting of a series of questions and other prompts for the purpose of gathering information from respondents. Questionnaires are commonly used in fields like market research, social science research, opinion polling, and customer satisfaction surveys. They allow researchers and organizations to collect data from a sample of people in a structured way.

 

3. Setting a Survey schedule for my UX final project

Date	Activity
3.11-3.18	1)    Determine test audience and goals. Finish redesigning a Chinese food ordering website.
3.11-3.18	2)    Start writing all the possible questions.
3.19-3.25	3)    Finish writing questions and review with a few people. 4)    Revise or rewrite as necessary. Write a report draft. Choose an online survey provider.
3.19-3.25	5)    Set up my survey using the service I chose. Pilot test, using both collection and tabulation software. Write report based on pilot test results.
3.19-3.25	6)    Rewrite questions based on pilot test feedback; review with a few people.
3.19-3.25	7)    Finish preparing online questions. Test survey for functionality under multiple conditions.
3.26-4.1	8)    Field the survey. When done, remove the survey site immediately and shut down data collection.
4.9-4.15	9)    Begin analysis.
4.9-4.15	10) Complete analysis. Begin report.
4.22	11) Complete report. Present to others, discuss, and note directions for further research.

 

4. Brainstorm Our Questions

• Keep the survey goals in mind and brainstorm every possible question we can think of that we want to answer with the survey.
• We can ask our friends or classmates for their opinions about the possible questions. And then share them our lists so they are not biased by our ideas. After that, we can see if any additional questions should be added to our initial lists.

 

• Two different survey goals are descriptive goals and explanatory goals.
• 1). Descriptive goals: aim to profile the audience in terms of personal characteristics like what they own or what they want. We do not try to understand how any of the characteristics affect each other.
• 2). Explanatory goals: explain people’s beliefs and behaviors by discovering relationships between their answers. For example, a descriptive survey would seek to know which features people use and what their average incomes are, whereas an explanatory survey would try to explain how the size of their income affects the features they prefer. Explanatory goals aim to find inherent relationships between characteristics.

 

• There are mainly 3 categories of survey questions.
• 1)Characteristic questions:

a). Demographic questions. These questions are about who the respondents are. Who is he/she? How old are they?  What do they do for a living? How educated are they?

b). Technological questions.   These questions ask about their digital technology setup and experience. What kind of mobile phone do they own? What is his/her physical& software environment like？

• 2). Behavior questions: outline how someone behaves.

a). Technology use. These questions ask users how they use the technologies they care about. How often are they online every week? What kinds of things do they use their cellphones for? What kind of computer experience do they have?

b). Usage. What product features do they use? How often do they use them? What are the reasons they come to the site? How long have they been using it?

c). Competitive. What other sites do they visit? How often and how long have they been using them? What features do they use?

• 3). Attitudinal questions: inquire into what users want and believe.

a). Satisfaction. Do they like the product? Does it do what they had expected? Are they able to do what they want with it?

b). Preference. What do they find most compelling about the product? What do they consider unnecessary or distracting？

c). Desire. What do they want? What features do they feel are lacking?

 

5. Write the Questions.

The close-ended questions provide the respondents with limited answers they must choose from. They can’t air their opinion the way they want, and their response is minimal.

On the other hand, open-ended questions allow the participants to respond as they wish, no holds barred. They are given the opportunity to share their opinion and their thoughts entirely.

• Close-ended questions: show most in the survey. They may be more suitable for the survey situation when there may be hundreds of or thousands of participants.
• Open-ended questions: appear most in interviews. They require much more effort from the person answering them and from the analyst. This is desirable in long interview situations with a few people. Also, they can provide answers in situations where researchers have no clue about how to write the question in a close-ended way. But they need to be used carefully and sparingly.
• The most common type of closed-ended question is the single-answer multiple-choice question. It has a range of choices for the respondent, only one of which may be picked.
• Another common type of question is the checklist. This question consists of a list of answers, any number of which can be chosen.

 

6. Some tips for designing questions:

1). Don’t make people predict their behavior.  People’s past behavior is usually better at predicting their future behavior than their statements are.

2). Don’t overload questions. Each question should contain at most one concept that we are investigating. Compound questions are frustrating for the respondent who only agrees with half of the question, and more complicated for the analyst who needs to infer the respondent’s perspective on both parts of the question.

3). Be specific. Avoid words with multiple or fuzzy meanings (“sometimes”, “around”, “roughly”, “any”). When speaking in units other than money, percentages, or other common abbreviations, make sure that the whole name of the unit is written out (“thousands” instead of “K”,etc.).

4). Never shut people out. Questions should always give people an option that they feel applies to them.

5). Stay consistent. Ask questions the same way every time. This means more than just using similar wording for similar questions. You should also strive to maintain consistency in meaning in the order of answer options and in the way the questions are presented.

6). Avoid extremes. Extreme situations rarely happen, and most people infrequently find themselves exhibiting extreme behavior, so avoid situations that require or imply the need for extreme behavior. (Avoid “every time”, etc.)

7). Make questions relevant. If people are confronted with a list of questions that don’t relate to their experience or their life, they’re not likely to finish the survey.

8). Use Likert scales. Likert scales are a familiar method of presenting multiple-choice answers. They consist of a statement or series of statements followed by a choice of 3, 5, or 7 options (most surveys use 3 or 5) that define a possible range of answers, including a neutral middle option. (“Rate the following aspects of the food websites for how interesting they are to you” .)

 

7. Edit and Order the Questions

• One way to maximize the number of responses to a survey is to keep it short. Most people should be able to complete the survey in 20 minutes or less.

Surveys that take more than 20 minutes begin feeling like a burden and respondents have to schedule time to do them. Since on average reading the instructions takes about 5 minutes and each question takes 30 seconds or so to read and answer. This limits us to about 30 questions total. Keeping a survey to 20 questions leaves us with a safe margin.

 

• The question order is as important as the wording. Question order should pace, focus, and selectively reveal information.

A survey is like a short story. The beginning grabs the readers’ attention, drawing them in. As they read the survey, they begin to get an idea of what kind of information the survey is trying to find out.

In the middle, big ideas are explored, and “twists” on the basic plot are introduced as certain avenues of inquiry are followed.

Finally, loose ends are tied up, and the survey ends.

 

4 parts of a typical survey
1)      An introduction	l  The purpose of the survey, l  Instructions for filling it out, l  The estimated duration of it, l  Contact information in case questions arise.
2)      A beginning	l  With teaser or interesting questions to draw the attendees of the survey in, l  Should not be demographic questions (which may bore people and ca n be seen as intrusive at the early stage)
3)      A middle	l  Alternating questions l  Questions should be grouped thematically (using small titles to categorize the questions)
4)      The end	l  With all the remaining demographic questions l  Providing an open-ended field for general response l  Reiterating the contact information.

 

8. Write the Instructions

1). General instructions

1)      The survey is important.	“We want to make… a better service for you. Your participation is very important to us.”
2)      What it is for.	“The survey is to help us understand the needs and desires of the people using…”
3)      Why people’s answers are safe.	“All of your answers are confidential and will be used strictly for research. There will be no sales or marketing follow-up because of your participation in this survey.”
4)      What the reward is.	“By completing this survey, you will have our gratitude and a 1 in 100 chance of winning a…”
5)      Who is responsible for the survey.	This survey is being administered for*** by …
6)      How long the survey is running.	This survey will run from March, 26,2024, until April, 1, 2024.
7)      Who to contact with questions.	If you have any questions or comments about this survey, you may enter them into the form at the bottom of the survey or mail them to*** at ***@***.

 

2). Individual question instructions

Question instructions should be simple and straightforward. Most close-ended questions won’t need special instructions, but open-ended questions should say exactly what kind of answers we want. For questions where we want a single answer, but people may feel that any of several options may be adequate, make it clear that they have to select what they feel is the strongest option.

 

• B.  Personal experience about an interview.

Currently, I have no experience in designing a questionnaire and conducting a survey, so I just want to share my thoughts about some do’s and don’t in doing an interview.

 

1. Choose the participants from different age groups, occupations, and genders.

If our surveys are not targeted at a specific age group, job, or gender, we should invite as many people from different groups as possible to achieve more convincing results.

Gender may make a difference! Last week, when I interviewed 4 Chinese about their experience with the food website. I found that 2 females had much better ratings than 2 males with rich experience designing the website themselves. Two men’s working experience may influence their judgment on the layout and structure of the website. However, 2 females may focus more on the aesthetic aspect of the color, images, and fonts. Just as Dr. Pullman said, I need to invite more women and men participants to test if different genders have specific preferences in website design.

 

2. Tell the participants that we have nothing to do with the product or service in advance.

As most of the participants we want to interview or invite to participate in a survey are our friends, acquaintances, classmates, colleagues, or even relatives, it is better to tell them beforehand we have nothing to do with the product or service itself. Otherwise, they may want to please us or say something nicer than it is because they want to be “good people” who do not hurt us instead of “good respondents” who tell the truth.

 

3. Better not to choose close relatives as the participants.

Last week, I invited my husband as the interviewee. Although he told me some of his true opinions and perceptions about his user experience with the food website, it was hard not to be judgmental towards his answers when they were not what I expected. We argued for some different ideas during the interview. If we invite some acquaintances with some social distance, it is easier for us to be more rational and respectful of their unexpected answers.

 

4. Designing questions is an ongoing process.

I told each interviewee the interview took them around 20 minutes. But after I interviewed the first 2 people, I found the interview with 16 questions took up almost 40 minutes. It is very significant for me to make the questions fewer and briefer. Just as Dr. Pullman said, “Your goal is to ask the least number of questions possible to learn what you need to learn”. I need to revise my questions and delete some questions with repeated ideas. Also, I need to order my questions with some grouped themes, such as “UI, UX, and Functionality” to make them more logical to answer.

 

 

Some useful links for designing the questionnaires (open-ended/ close questions):

1. https://www.invespcro.com/blog/open-ended-questions-and-closed-ended-questions-what-they-are-and-how-they-affect-user-research/
2. https://blog.hubspot.com/sales/the-art-of-asking-open-ended-questions
3. https://copyhackers.com/2022/03/what-is-an-open-ended-question/