How to Write Good Survey Questions
How to avoid biasing your questions and annoying your respondents.
Sponsor: Audible.com the Internet's leading provider of audiobooks with more than 100,000 downloadable titles across all types of literature, including fiction, non-fiction and periodicals. For a free audiobook of your choice, go to AudiblePodcast.com/GG.
Today we’re going to talk about a kind of writing that you might not think of when you think about writing. At least, I don’t. It’s questionnaires and surveys. Whether you’re doing scientific research, investigating the market for a new product, or just having fun on your blog or Facebook page, you need to think about how to write your survey questions if you want them to give you the best information possible.
I looked for some references for writing survey questions, and the main thing I discovered is that there is so much to think about that if your survey is really important, you should consult an expert. Even so, experts agree on a number of points that should give a more professional feel to the surveys you do yourself.
Here’s an example of a biased question: “How much did you like the movie?”
I consulted three websites for these tips, and I’ll put links to all three of them on the transcript. The website HowTo.gov is intended for government workers who want to assess customer satisfaction to improve how services are delivered. I used the ”Basics of survey and question design” page from their section on collecting feedback. I also went to the nonprofit website Science Buddies, which provides guidance and science project ideas for K-12 students and teachers and has a page on writing survey questions. For the information on this page, they credit the textbook Marketing Research by Parasuraman, Grewal, and Krishnan. Lastly, I visited the website Creative Research Systems. This is a company that wants to sell you a software package for creating and processing surveys. However, they have a very informative section on designing survey questions, which agrees with the advice given in the other sites I’ve mentioned. All of these websites give much more information than I’m going to give here, and I recommend visiting them when you can. Each site has various details that the others don’t have, but they also agree on a lot of points. Those are the points I’m going to present today.
Define Your Survey Goals
First, you need to know what kind of information you want to get from your survey. If you have only a vague idea of what you’re trying to find out, your questions will be vague, too, and so will your answers.
Put Easy Questions First
As for ordering the questions, you should put them in a logical order, and group questions on similar topics together. If possible, easier questions should come earlier in the survey. Again, this makes it easier and more pleasant for respondents to take the survey, which increases the likelihood that they will actually finish it. In oral surveys, this also helps the interviewer build rapport with the respondent.
Put Difficult or Sensitive Questions Last
Conversely, put the more difficult questions near the end of the survey. If respondents see a tough question right at the beginning, for all they know, all the questions could be that difficult, and filling out the survey starts to look like a real hassle. But if they see it at the end, they may put in the effort since they know they’re almost done, or because by this point they like you and trust you.
Even if they quit, at least you will have most or half of a survey to analyze, instead of none. This advice goes not only for questions that are just difficult, but also for questions that are more sensitive, such as questions about income level or ethnicity.
Be Careful When Giving Respondents Choices
The next few tips have to do with structured questions, that is, questions in which you provide a choice of answers. Examples of structured questions include multiple-choice questions, or questions asking respondents to rate something on a numerical scale.
Cover All Possible Answers
The experts agree that in a multiple choice question, the choices should cover all possible answers. Sometimes, this will mean including an option for “Other,” or “Don’t know,” or even “Don’t wish to say” for sensitive questions. Not only will this get you more accurate data, but it builds trust. If respondents feel you’re trying to make them give you an answer they don’t agree with, they may just skip the question, or stop answering questions altogether.
Make Sure Answers Don’t Overlap
In addition to providing for all possible answers, the choices in a multiple-choice question with just one response allowed should be mutually exclusive. They shouldn’t overlap. For example, if a question asks which kind of food is your favorite, the answers shouldn’t include both Thai and vegetarian, because some food is both. If someone’s favorite food is vegetarian Thai food, which response is appropriate?
Ask About One Thing at a Time
You should avoid “double-barreled questions,” that is, questions that ask about more than one thing at a time. For example, if you instruct a respondent, “Please rate your satisfaction with the service and food quality during your visit,” you don’t know what kind of answer you will get. Will the respondent rate the service and overlook food quality? Will he or she do the opposite, or maybe just report whichever rating is lower, or higher? Instead, break this into two questions, one about the service and one about the food quality.
Avoid Biased Questions
For any kind of question, you should make sure it is not biased to make the respondent more likely to give a particular answer. At least, this is what you should do if you want as accurate a reading as possible. If, on the other hand, you’re a sleazy politician who just want a survey to make your candidate look as good as possible, or the other candidate to look as bad as possible, then by all means you should use loaded, emotional terms, and phrase the questions in ways to get the answers you want. And you should go away. But if you’re listening to this podcast for tips on better writing, you’re clearly not one of those people.
Here’s just one example of a biased question: “How much did you like the movie?” That question will bias respondents toward a positive response, even if your answers include a “Not at all” choice. “How did you feel about the movie?” is more neutral.
There are so many ways to bias a question that I can’t go into them all. Aside from the words you use, the order in which you present the answers to the questions, and even the order in which you present the questions themselves, can affect respondents’ answers.
In part because it is so easy to bias a question, the sources agree that you should test your survey before you administer it for real. Have colleagues read it, and have a handful of people take it so you can see where they get confused, or where any other problems come up.
Finally, you should thank your respondents for helping you!
Again, there are many more tips to keep in mind when you create a survey, and I recommend checking the sources I used here for more information and more details on specific types of survey questions. However, the few tips we covered today should help you to avoid some of the most common mistakes.