The biggest challenge for a Data Driven culture is ensuring good quality data is collected consistently for every need. Among other options, Surveys have proven to be a trust friend to researchers. But anyone who tried their hands at online surveys knows too well that you could end up with useless data, or worse, very little data to base your research on. In this post we will do a complete run down of the online survey process – the what, the why, and importantly, the how.
1. Firm up on your goal early in the process
Research is both a consuming experience and a journey that can easily go off the track. It’s quite common to get distracted by new information you uncover and get tempted into chasing something totally different to what you started off with. Spending enough time on the goal early on gives you the conviction to stick with it no matter what temptations arise in the due course.
Now the goal may not be crystal clear to begin with. Spending time on existing knowledge around the problem at hand may help you sharpen the definition and arm you with enough to lift off your research process with a good thrust.
2. Narrow down on your audience
Research activity is hungry for data, but good research eats healthy and exercises well. We’ll get to the exercise in a bit, but let’s talk about the data first. Quite often the quality of data suffers from the point of creation itself. Make sure you understand your audience early in the process. You could use the following pointers:
- Audience may not always be coming from a single demographic – age, location, ethnicity, etc. It helps to list down the broad segments that you would be going after. You could even splice your survey to compare two segments and see how they react (plug: EV allows you to do that btw).
- In general your audience population is too big to go after. Pick a sample that is a good representative of the population. That’s easier said than done, but you have to keep an eye on it as you execute your survey.
- Have a rough estimate of the number of responses you want. There isn’t a gold standard here, but depending on the area you are researching on, you could arrive at something by looking at prior work by others. If you are not sure, go with at least 30 responses. There is a statistical significance to 30 that we won’t discuss here, but you could learn that from ‘the not so little book of statistics‘.
We are the same.
We are unique.
- Knowing your audience allows you to speak their language and be sensitive to their cultural and emotional context. You could arm yourself with these as you create your questions, and at the least you don’t end up offending them.
3. Choose your Questions wisely
This one may seem obvious. After all questions are at the heart of a research activity. For the same reason, it’s good to be aware of the nuances of good questions that can make your survey great. Here are some of these nuances.
- Taking surveys is clearly not on the list of anyone’s favorite ways of spending time. It’s helps to both keep the question count to a minimum by making sure each question comes out with its own mini objective. Once you create the survey, question the existence of each question. Ask if removing a question has a real risk on the bigger objective. The urge to play it safe generally leads to questions that don’t offer any real analytical value, but make you feel covered from all angles. Be firm and get rid of any such questions. It helps both keep the survey short but earns you the respect from your audience for the effort you have put in to keep things short and focused.
- Remember short is sweet. Consider using simple language with fewer words.
- Be direct in your questioning. It helps clarity and gets you the intended effect. Beating around the bush may lead to the question being misunderstood.
- Don’t go overboard on making the survey short by combining multiple questions into one. You could easily end up confusing the respondent and worse, get them to quit midway.
- Be genuine in framing questions. Avoid loading the questions with opinions in a way that leads people in a certain direction. It is not only unethical but also defeats the very purpose of research.
- At times a simple Yes or No may seem to do the job. But think of the possibility that your audience may not be in a position to go with one of these. If it is possible that there are shades of grey between a Yes and a No, offer the choice. Scales like the one below help bring out this subtle yet crucial data point.
- Pick the right question types depending on the number of respondents and the importance of the information being gathered. If the number of respondents is low, you could review the responses to open ended questions and draw deeper insights. However, the same may not be feasible with a large audience and it may be better to present more closed ended questions.
- It pays to think of how to analyze each question ahead of time. That way you can improve the question, and establish its connect with the larger objectives.
- Try forming connections between questions, so you know in advance if some questions need to be taken in a certain sequence. This helps avoid a possible feeling of incoherence as the audience respond.
4. Dynamism is an important trait for a good survey
Having good questions on your survey is like getting the casting right. Now, even if you have Al Pacino, Robert De Niro, and Leonardo Dicaprio starring together in a film, you need a Martin Scorsese to direct the movie for it to be a hit with the audience. Now, Scorsese doesn’t have much interest in your research project so pull up your sleeves and get ready for some lights, camera, action.
- Qualify your audience early on. Include questions that help you decide if the respondent is even a fit. As an example, if your survey is about saving habits and your respondent says he has never earned money let alone save it, you should know too well to not let them go on with the rest of the survey (plug: EV allows you to use logic jump to achieve this).
- Be mindful of the order of the questions. Like you pace an important conversation, start off with questions that are relatively simple and don’t need too much thinking on the respondent. As they pick up momentum, questions that need consideration don’t feel too heavy.
5. Distribution – test drive, timings, calibrate
Referring back to our movie example, a good movie also needs an equally good distribution strategy to succeed. Similarly, a survey needs to be distributed smartly so as to get good quality data.
- You could start by conducting a test drives to vet your survey. This generally helps you fix any potential issues (don’t under estimate this even if you have a great eye for detail).
- A test drive also helps in calibrating the survey from a duration point of view. When you review how a real respondent behaves to your survey, there is generally some tweak you may want to make. You could either feel an urge to alter the length of the survey, the options, and even the language.
- When you feel confident with the test and the adjustments post the test drive, you could roll it out to the larger audience. Even here, you could keep an eye on which segments are more active versus others. Depending on how skewed the participation is among segments, you could push it a bit harder in the segments that did not participate enough to get an unbiased input.
6. Analyzing the data collected
This is the part of the process that is filled with most action. The sooner you realize the importance of analysis in your research journey, the better off you are. Being conscious of the need to research helps you to effectively run the activities leading up to research.
You could test the responses early on for their usability and make tweaks to the survey questions. You may change a few questions from open ended to closed ended. You could drop a few questions if you don’t see them add value, or add some a few questions to fill up missing pieces of the puzzle.
Here is how you would go about the analysis:
- Start off by preparing the data for analysis. This post on cleaning up your survey data helps you do just that.
- As a next step, categorize the data collected as quantitative and qualitative, and apply relevant techniques for analyzing each type.
- Qualitative data generally includes opinions, subjective inputs, and other ling form responses.
- Analyzing qualitative data can get tricky and this is where you need to plan ahead of time. Too many responses make this both a laborious activity and one that is hard to draw patterns out of.
- Quantitative data on the other hand is generally easier to fit into tables. For example, data of families in a community – number of people in each family, type of car they own, ethnicity, highest education of male and female members, occupations etc.
- You can refer to our free books on Statistics for more on understanding data types and analyzing quantitative data.
All the hard work you put into the research process culminates into a report. This is where you wear the hat of a storyteller. You draft the entire journey starting with the motives behind your research activity, the hypothesis you came up with, the reasons for picking the specific segments of audience, rationale behind the questions employed and the way they are presented, and importantly the analysis performed the the inferences drawn.
Here are a few tips on coming up with a winning report.
- Keep an open mind to the outcomes of your analysis. Many noob researchers sink into a feeling of disappointment when the finish the analysis.
- Picture your audience as you draft your report. Maintain a fine balance between sharing facts and narrating a story. This post on telling a compelling story based on your research should help you get a start.
- Use visuals to add to the experience of consuming the report. They help generate interest, and highlight the important aspects of your report. You don’t have to limit yourself to charts, feel free to experiment with infographic style visualizations.
Research is quite a consuming experience and can get quite daunting on new researchers. Most of the above inputs should help you shine with your next research project whether you are running online surveys or not.
Wishing you the best for the next one!