Should you DIY your next survey?
When we talk with clients about their market research projects, most say they are juggling three things: time, money and data quality. When you’re squeezed on time and money, online DIY surveys – such as Qualtrics, Google Surveys and Survey Monkey – may seem like more attractive options than using a research partner. But the benefits may not outweigh the risks. Here is what you need to know and consider about using a DIY approach for your next survey.
When you need immediate information …
Many DIY survey providers talk about giving you fast answers, but their ability to deliver on that promise depends on who you are trying to survey.
- B2B audiences typically respond much slower than consumers, especially to online surveys.
- The panel/database quality can affect response times. How engaged are the potential respondents?
- Finding niche audiences may also take longer, based on how much targeting can be done on the panel/database. For example, are you looking for vascular surgeons but can only target on “surgeon”?
Just because it’s DIY does not automatically mean you’ll get survey answers any faster.
When you are tempted to monitor the data as it’s collected …
Some DIY surveys enable you to track data real-time. This might enable you to perform a preliminary analysis and gather initial insights. While this instant gratification may be tempting, here’s why SMS recommends against providing real-time or preliminary data.
- Differences and trends that emerge among the first 30 respondents may become less strong once all the responses are gathered (e.g., among 150 completes).
- This volatility in the data can introduce bias in analysis and interpretation. Because you’ve already been exposed to an insight during data collection, you may hyper focus on it and miss other, new insights.
- Differences between preliminary findings and final findings, even if those changes are only slight, open the door for stakeholders to question the validity of your data.
When you are concerned about costs …
If saving time isn’t a huge differentiator when choosing between DIY or a traditional research firm, then what about costs? DIY surveys may look cheaper on paper, but not necessarily in terms of your ROI.
Consider why they are so cheap. DIY surveys cut out other costs that traditional market research firms cover, such as survey development, database cleaning and project management. If these can be covered in house with confidence, then DIY surveys can be a great tool. Emphasis on: with confidence.
To determine if you’re up to the task, consider these questions (and be realistic):
- How well-defined is the audience you are targeting? For example, how good is your database or the provider’s panel (if available)?
- How much time do you have to invest in your DIY survey project?
- Consider not just initial programming but testing and addressing bugs/issues. A basic rule of thumb: Estimate however long you need for database set up programming, then double it.
- Are you available/prepared to be frequently interrupted from your daily tasks to regularly monitor completes, send out reminders/invites and answer any respondent questions?
- What level of analysis and insights do you need? How complex are the research questions you need answered?
- Do you only need to know how satisfied a customer was with a recent purchase, or do you need to know what attitudes, opinions and behavioral norms influence the way customers interact with your brand?
- If the latter: Are you aware of advanced analytical models and can you perform these analytic techniques?
The final, greatest question: how important is this research?
How much buy in do you need from key stakeholders? Are you ready to defend your DIY survey results if they are not as expected?
DIY surveys can pose a serious risk when your goal is to gather reliable market data for better decision making. All results and insights hinge on two key areas of the research: questionnaire development and improper sample selection. You can read more about the challenges and considerations in survey development – plus get some tips – in our upcoming blog post, “Survey Bootcamp”.
DIY surveys can be a great option for those with both research and programming expertise. But some of the “pros” may not be as useful as they seem and may not outweigh the risks.
I can help you decide whether a DIY approach is right for you. Contact me at firstname.lastname@example.org.