The current COVID-19 crisis has effectively halted in-person research. Whether you’re a client-side researcher who still needs these insights or a market research agency whose business has been curtailed, there are now opportunities to move this work online. In this Research Note we explore both the data collection and sampling implications for migrating offline work online and offer perspective on what to keep in mind when creating new research design.
Quick Fix or Big Change? Will Things Ever Get Back to “Normal?”
The most frequent question we have been asked is what changes should we make to compensate for the current situation? These questions typically come from quantitative researchers. Upon further probing, we identify two concerns. One is how to properly adjust their research systems in light of any behavioral and demographic changes caused by the pandemic. The other is whether to think of these changes as temporary or permanent in nature. Quite often their thoughts turn to changes in sample quotas or weighting.
Offline researchers are obliged to ask a more fundamental question: should we change methods altogether? After all, there were reasons for choosing an offline methodology in the first place. The duration of the global lockdown and its potential aftereffects are only part of the issue. The bigger concern is whether the research can be performed effectively online.
As a general principle which should surprise no one given our business, we have always advised researchers of all stripes to be actively exploring the ways in which technology is changing our industry. Given the crisis, it makes even more sense to do this as it maximizes one’s options.
Why We Choose Offline Data Collection and Sampling
There are two essential elements to any insights project: data collection and sampling. When we choose a given methodology, we evaluate its implications for these elements to determine whether it is fit for purpose. Sometimes we make choices for positive reasons; there is something advantageous about data collection or sampling that leads us to choose a given methodology. Sometimes our reasons are negative; we may prefer another methodology but there is something about the data collection or sampling that prohibits that choice.
Historically, the reasons for choosing offline data collection have been positive in nature. We prefer offline methods mainly because they allow us to observe people in greater detail and make it easier to understand why people do things. In–person interviews allow us to probe complex subjects. Intercepts allow us to catch people “in the moment” and interrogate them while an experience is fresh.
Conversely, the sampling-related reasons for choosing an offline methodology tend to be negative. For years, the economics of traditional panels meant that smaller countries were often inaccessible. Limited targeting data made it costly to run low incidence work. In these cases, intercepts could be justified to shorten the odds of finding the right people given the already high costs.
Moving Online: Data Collection Considerations
The variety of offline methods (and our desire not to play favorites) prohibits a full listing of them here, however there has been a veritable explosion of new platforms that have enabled online data collection for work that was once done exclusively offline. From online focus groups and interviews to portable/digital neuroscience techniques, from basic functions to AI-driven analytics, there is plenty to discover. Your favorite MR association (ESOMAR, Insights Association, MRS, or others) can help identify potential partners. We can also recommend searching the numerous blog articles that appear every week in publications like Research Live, Quirk’s, GreenBook, and Research World (ESOMAR).
Evidently, any new data collection method needs to be tested prior to use. A complete test from start to finish in “full production” conditions is essential to understand not just how the platform works, but what processes may need to be in place to use the platform and what the “output” looks like.
Most importantly, researchers will need to pay attention to how research subjects behave. Online execution has the potential to yield less attentive responses. There is manifestly less control over the respondent’s environment. This is not to discourage researchers, rather to suggest vigilance. Above all, researchers should be suspicious of comparatively cheaper pricing. While lower costs are to be expected to some extent, significantly cheaper pricing is likely to lead to fewer thoughtful responses and less patience on the part of the respondent.
Moving Online: Sampling Considerations
For years, face-to-face interviews (or even CATI during the online era) have solved problems of availability. The Internet was hardly ubiquitous at first, and more tenured researchers will remember— amongst other things—long debates about the “Digital Divide.” In the digital age where powerful computing devices are cheap, portable, and supported by infrastructure that was unimaginable when telecommunications required wires, the reach and diversity of audiences available for sampling has skyrocketed. Moreover, the idea of making a massive investment in setting up new panels in new countries is almost quaint. Now, sample supply is predominantly programmatic in nature. Application Programming Interfaces (APIs) transmit specs and availability of demand and supply in an instant.
Technology has also allowed suppliers significantly reduce sample waste and find the right person for the right survey at the right time. Automation has led to much deeper profiling of behavioral, and more sophisticated suppliers use algorithms designed to estimate a potential respondent’s probability of qualifying and completing a survey to maximize the all-important conversion rate, defined as the number of completes divided by the number of starts.
This is not to say that every audience is now available in numbers online. Furthermore, the ability to access an audience does not obviously mean people will be willing to participate, but rather that finding sample of even niche audiences is much less difficult today than even a few years ago.
Offline researchers will need to be careful when setting up sample designs with those accustomed to doing online work. A default setup will be whatever the supplier considers to be “nat rep” (nationally representative), which varies from supplier to supplier and occasionally has nothing to do with true national representation. It may be the case where the work does not require this level of national coverage. The supplier will be mainly focused on length of interview (LOI) and incidence rates (IR). These metrics are the indispensable elements of any online quotation, but they only partly describe offline work. Make sure the nature of your work is clear not just to the salesperson, but to the project management team as well.
Above all, offline researchers should initially resist the temptation to increase the sample size given the cost effectiveness of online sampling in relation to offline work. Test first to make sure you are getting the information you need. And remember: if you have been content with small convenience samples up to this point in time, ask yourself what will change if you decide to do more.
The “Truth” Will Change
Seasoned researchers understand that we can get different answers to the same question depending on the methodology we use. It stands to reason then that switching methodologies will surely result in a new version of “The Truth.” The hardest thing to communicate to non–researchers is that different methodologies will, because of imperfections in BOTH methodologies, yield different results. It is harder still when an existing methodology has been socialized and accepted as Truth. People will look suspiciously at the new data. Managing this sort of change is a project in and of itself which we will cover in our next Research Note.
Conclusion
If COVID-19 has you contemplating methodology changes, you are not alone. At minimum, exploring these changes fully increases optionality, and starting sooner rather than later means more time for reflection and testing. If there is a bright side to the situation, it is that the steps one must follow are simple and unambiguous, even if there are complex choices to evaluate. The Cint team is available to provide expert advice to clients wrestling with these issues.
Let’s talk
www.cint.com/start
Written by: JD Deitch (COO at Cint)
View our other Research Notes:
- Research Note #3: Moving Your Offline and F2F Research to the Internet?
-
Research Note #2: Is COVID-19 impacting Market Research Supply and Demand?
-
Research Note #1: Monitoring Changes in Consumer Behaviour Due to COVID-19
