Research Note #4: Is Respondent Quality Declining During COVID-19?

As the industry’s largest sample exchange, we at Cint have good visibility into the oceans of people who participate in online research. Despite the COVID-19 crisis, respondent levels remain high. Through our panel management platform, we saw huge increases in registration and activity among our 4000+ partners during the lockdown. With all this interest in online research during these unusual times, one of the more frequently asked questions we receive is “How are we maintaining quality?” This is the topic of our fourth Research Note.

  Every sample supplier says they take steps to ensure quality. What they do varies considerably, however, in both form and substance. In this age of programmatic delivery where automation is driving both faster execution and faster deceit, the details matter a lot. For those of us who operate platforms and exchanges, the standard must be higher. We cannot content ourselves to reporting fraud levels hoping clients make good choices amongst a patchwork of different supplier initiatives. At Cint, we strongly believe it is our responsibility to actively ensure quality for every transaction in real-time, both for the health of our platform and our broader industry. That means clearly defining the term quality, robustly measuring it, and having the best people, products and technology so that every transaction we process is subject to the same state-of-the-art techniques. WE are accountable for quality on our platform. Here’s how we do it.  

Our Quality Charter

We cannot hope to ensure quality unless we define what quality means and can measure its indicators. We identify five components of quality at Cint:

1. Respondent Quality, or the extent to which respondents are real, unique, engaged and representative
2. Supplier Quality, or the extent to which our supply partners are providing high-quality respondents
3. Buyer Quality, or the extent to which our clients are providing compelling, high-quality studies
4. Operational Quality, or the extent to which we execute well and deliver on our commitments to clients and partners
5. Product Quality, or the extent to which our products provide industry-leading and future-proof solutions

Respondent Quality—and by extension Supplier Quality—is the topic that interests most of our clients.  

Respondent Quality

We consider a respondent through four lenses:

1. REAL: Is the respondent real and not fraudulent?
2. UNIQUE: Has the respondent participated once and only once?
3. ENGAGED: Is the respondent providing accurate and complete answers?
4. REPRESENTATIVE: Do our respondents, collectively, exhibit demographics and behaviours consistent with a known population?

We assess our performance on each of these elements through different indicators. For the most part, the first three (REAL, UNIQUE, and ENGAGED) are most easily assessed by looking at the number of completes that are rejected for each of these reasons. The industry calls these “reversals.” Representativity is evaluated through benchmarking studies. Reversal rates are the currency of quality in the online sample ecosystem. We become aware of quality issues in two ways. One is when our system detects a problem. The other is the rare and regrettable occasion when we are alerted to a problem by a client. Over time, we have aggregate measures of these events so we can detect where we need to improve. Most commonly we look at these measures by country and supplier.  

Combatting Fraud

With any “flare-up” in fraud, our dedicated quality team springs into action. Working closely with our commercial teams, partners and clients, we typically have a report and a remediation plan within 24 hours of the incident. Depending on the issue, our actions could take many forms:

• We will always ban panelists who are fraudulent to ensure they can’t use the same accounts again.
• We may stop accepting supply from certain suppliers by countries or implement delayed payment for respondents in extreme cases.
• We will “turn up the dial” on our AI-based fraud detection products to be more stringent in limiting flow.
• We will provide the latest list of fraudulent respondents to our technologists to train the machine learning algorithms on the new fraud.

Our investigations are always data-driven, but responding to these issues is not always straightforward. While we can clearly see the fraud, the primary challenge with any fraud detection system is balancing detection of real fraud versus “false positives.” We could reduce fraud by cranking up our algorithms to maximum sensitivity, but this would reduce sample flow to a trickle. Our Key Performance Indicators (KPIs) help us monitor and manage this issue.  

Measuring Fraud

We will never eliminate fraud. Fraud is endemic to the broader marketing ecosystem in which we operate and from which every supplier recruits research participants. Moreover, the technology that allows us to do our own work more efficiently is the same tech that fraudsters use to attack suppliers and data collection platforms at scale. Finally, given the issue of false positives, we are always asking ourselves one all-important question: How much fraud should there be? Or we could put it differently: How much is too much? If this sounds like a crazy question, consider that the pursuit to purge all fraud would be impractical and massively expensive. Once we understand this, we can at least talk about fraud in relative terms. At Cint, we aim to be among the best in the industry. We will never know exactly where that is (others don’t typically share data), but we have a good enough idea from our clients and partners where we need to be. Below are our platform reversal rates for the past six months.

• Nov 2019: 4.3%
• Dec 2019: 3.7%
• Jan 2020: 3.0%
• Feb 2020: 3.8%
• Mar 2020: 3.8%
• Apr 2020: 2.3%

We are quite proud of these figures and of the considerable effort our team has put forth, notably over the past year (especially on the topic of redirect fraud), to stay one step ahead of the bad guys. They also indicate that we have been able to maintain a high level of quality despite the number of new registrants and greater platform activity. That said, these figures are platform averages. The truth is that there is never really a tiny trickle of fraud in roughly equal proportions in each and every project. The vast majority of projects go along just fine with quite minimal levels of fraud. Then, out of nowhere, one just gets crushed. These are awful to live through as they create justifiable concern for everyone involved. But they are unavoidable. What becomes important is quick discovery and remediation as discussed above. It is equally important that we make sure are improving our systems to prevent—to the extent possible—the problem from happening again.  

Proprietary Technology and Intellectual Property

Having metrics does not mean anything if you do not act on them. In our view, that means acting in real-time to prevent the issue from happening. Cint uses both proprietary and off-the-shelf products to manage respondent quality in real-time. Its acquisition of P2Sample has added significant capability and expertise in this regard. From the moment we see a respondent, we track his or her every move. We capture personally identifiable information (PII) and profile the panelist deeply. We use this data and other proprietary data to ensure each respondent is unique, creating a proprietary “digital fingerprint” that goes beyond machine and connection characteristics. These digital biometrics are fed into our artificial intelligence-based fraud systems so we can detect evolving threats in real-time. AI also allows us to evaluate attentiveness and response quality. We have invested considerably in AI and see it—and the machine learning elements of it—as essential to the health of our platform and our business. Our work on AI goes beyond fraud and includes techniques to match individuals with surveys they are maximally likely to complete, which is the right outcome for everyone, especially the respondent. There are plenty of other, more traditional techniques we use as well. From email and postal address verification to speeding traps, we deploy a host of measures to ensure we have real, unique, engaged, and representative respondents. All in all, it means better data for our clients.  

Proactive Programs with Suppliers and Buyers

Much of the discussion above has revolved around our reactive approach to quality. We work proactively with our supply partners and clients on a variety of subjects. On the supply side, we share data and techniques to show suppliers what we see and then root out trouble. Our work is not limited to dealing with suppliers. As we all know, buyers are in equal measure responsible for quality. We aim to share everything from field statistics to respondent ratings to panellist support tickets from poor performing studies with our clients. While we can’t force them to edit their surveys, we can help them understand how things like late terms and high dropout rates directly impact the amount of sample their studies receive.  

Thought Leadership

Finally, a vital element of our quality program is sharing what we know with the industry. The COVID-19 crisis has only accelerated our efforts to provide a strong signal to insights professionals of all backgrounds. We publish articles like these on our blogs as well as on respected industry and association websites. The pandemic has not stopped us from participating in conferences either, even if these conferences are now online! We will continue to talk honestly and forthrightly about all sorts of topics, from automation to fraud, from the minutiae of sampling to the bigger picture principles of research.  

Conclusion

We hope this Research Note, our fourth since the word Coronavirus entered our vocabulary, has made it clear how seriously we take the subject of quality. We know quality issues can crop up. Our techniques will have to evolve. But we have the right framework and approach. The health of our platform and our industry depends on us acting responsibly. Quality is absolutely one of the areas where we live by our values every day. We are accountable for quality on our platform whether that is the quality of responses, of inventory, or how the system performs. We regularly collaborate with our clients and partners to troubleshoot problems and improve quality. We create innovative products that actively and uniformly allow us to manage quality platform-wide.  

For more information get in touch with us:

cint.com/start

   

---

Written by: JD Deitch (COO at Cint)

---

View our other Research Notes:

• Research Note #3: Moving Your Offline and F2F Research to the Internet?
• Research Note #2: Is COVID-19 Impacting Market Research Supply/Demand?
• Research Note #1: Monitoring Changes in Consumer Behaviour Due to COVID-19