CAWI
Here are some risks you must pay attention to and how we avoid them:
Risk:
There is no respondent, the answers are fictional.
There is a risk of studies being completed by bots, not by real humans.
Solution:
We validate each panelist through several methods: each person must have a name, phone number, email address and, most importantly, a valid Romanian bank account. Without a bank account they cannot receive any reward, thus ensuring that each completed questionnaire is submitted by a real person.
Quality criterion in the RFQ phase:
Ask how the provider validates that each panelist is a real human.
Risk:
Respondents are real, but they answer multiple times to your study.
Given the chance, some respondents would answer several times to the same questionnaire, just to earn multiple times the reward.
Solution:
We make sure that each invitation link is unique and can be completed only once; if the respondent finishes or is screened-out, the link becomes inaccessible for further completion.
We validate that each account in our panel has a different phone number, email address and bank account.
Of course, there is still a possibility that someone could create two accounts using different phone numbers, bank accounts, and email addresses. For these situations we constantly monitor interviews that have similar IP addresses and similar demographics and we invalidate those interviews.
Quality criterion in the RFQ phase:
Ask if the links have a unique URL that is sent to a panelist with a unique ID and each link can be completed only once. Ask the provider how they verify that a panelist does not have multiple accounts.
Risk:
Respondents are reward seekers and not interested in giving thoughtful answers and opinions.
This risk cannot be mitigated completely. There will always be panelists who simply want to gather as many rewards as fast as possible, some of them almost making a job out of this: hunting for each panel, answering questionnaires fast, lying to qualify for as many studies as possible.
The objective is to reduce this risk as much as possible, and one solution is through how these panelists are recruited.
Solution:
From our experience with panelists throughout the years we have learned that recruiting panelists through online ads will lead mainly to people who are reward seekers.
That is why for the past years we have stopped any online ad, and we are actively recruiting only through CATI (computer assisted telephone interviews). These are people who accept to participate in CATI interviews without receiving a reward, and who afterwards we invite to our panel. These respondents sincerely want to express their opinion on different subjects and do not hunt for additional income sources by enrolling in all panels available on the market, thus ensuring higher data quality for each study. We are fortunate to have a very efficient and high-quality telephone interviewing system and do many random digit dialing interviews (for instance the radio audience survey, the EU monthly consumer survey), ensuring a steady source of panel members only from this source. Practically we talk to our panel members before inviting them to join – there is no better proof they are real 😊
Quality criterion in the RFQ phase:
Ask through what method the provider is mainly recruiting panelists.
Risk:
Respondents are lying to qualify for the study and earn a reward.
Some panelists lie during the screener section of the questionnaire, just to qualify for the study and earn the reward. This might happen because some respondents are simply trying to earn as much money as possible from these platforms, or because they get frustrated by receiving invitations and being disqualified repeatedly. Which is somewhat understandable, imagine you are invited to several studies, for each you dedicate several minutes to the screening part and each time you are screened out without anything to show for your lost time.
Solution:
There are mainly two solutions to this problem:
- Trying to set up traps in the screening part of the questionnaire:
- Add questions that are not about the study subject, so that the respondent does not learn too early what the theme is and thus know in which direction to lie.
- Add contradicting answer options, or options that are very unlikely to be checked together and disqualify those who check them.
- Add fictional answer options (e.g. brands) and disqualify those who check them.
We constantly use these traps, and often they are effective, but there are some risks embedded in this approach: first, panelists will learn these tricks and try to avoid them, and second, they will become fed up and leave the panel, thus in time remaining only with the ones who desperately want the rewards and are willing to grind for them. From the get-go this approach is flawed, by relying simply on punishments and building obstacles.
- Rewarding the panelists for screened out and quota full questionnaires:
- This solution is simply better from multiple standpoints: it relies on mutual respect, not on punishments, it assures a longer-term relationship with the panelist and discourages lying simply because they know they receive a reward for their time regardless of the questionnaire outcome. In a repeat study after one year, we obtained 58% from such panels, compared with approx. 15% from large international panel suppliers. The difference is quite remarkable!
- We strongly believe in this approach, of course implemented alongside the first one, for better overall protection against this risk.
Quality criterion in the RFQ phase:
Ask the provider to offer incentives for screen-outs and to implement trick questions and options in the screening part of the questionnaire.
Risk:
All respondents are rewards seekers, thus not representing the intended population
This is closely linked to the one above and is about the incentive paid. If incentives are too small, insultingly small, any normal person will get out of the panel. And if such habits of insultingly small incentives continue, there will be no normal person in the panel.
Solution:
Ask the panel provider how much they pay as incentive and evaluate if it guarantees that there are normal people in that panel.
Quality criterion in the RFQ phase:
The provider should be transparent about the value of the incentive paid to respondents.
Risk:
Panelists are instructed to lie.
Hopefully, this happens rarely, but we have seen some real cases. This mainly happens with low incidence studies where panelists receive a pre-qualification questionnaire that allows them to see what characteristics are required because only those specific answers are available or just receive instructions for what is needed to qualify. Afterwards they are directed to the actual questionnaire where, if they remembered correctly what they have checked before, they can qualify for the study even if they are not the intended target.
Solution:
The solution is simple: not implementing any pre-qualification questionnaire whatsoever. And, additionally, delivering the client the entire database, including screen-outs, so that it is clear the recruitment was not forced.
Quality criterion in the RFQ phase:
The provider should not implement any pre-selection questionnaire, not inform respondents of the qualification criteria, and should not allow any subcontractor to do this. These get found out if you look for them, as many panel members receive invitations, sometimes tens of thousands, and some may know the study owner.
Ask for the full database of answers, including screen-outs. If, for example, you estimate a 5% incidence for your target, then successful interviews should represent only around 5% of the entire database with the screen-outs included.
Risk:
Unserious and careless answers due to low incentive.
Usually, clients do not think about the incentives panelists will receive for completing their study. They leave this decision completely to the provider, without actually realizing how important that incentive is for obtaining data they can actually rely on.
Do not delude yourself that if a respondent accepts to complete a study knowing from the beginning the incentive, then everything is fine.
First, a low incentive from the start discourages some people and your sample will become biased. And secondly, those who accept are unlikely to provide their best effort: imagine someone asking you to give their task your full attention for 10 minutes, only to receive 10 cents.
Thus, by accepting a lower price for the study and by not asking about the incentive planned for the study, you risk bad data.
Similarly, there is a risk if panelists are not rewarded for each study, but they simply participate in raffles and lotteries – this approach lowers their willingness to give serious answers.
Solution:
We are transparent with how much we pay respondents for each study, and this payment is non-negotiable. We do not reduce the incentive based on the client’s budget, rather we reduce the sample size or the questionnaire length to accommodate the budget. We do not reduce the incentive because we strongly believe in the necessity of a healthy panel that produces reliable data.
A fair and reasonable incentive differs from country to country. For Romania we consider 0.5 lei per minute as a reasonable incentive.
Quality criterion in the RFQ phase:
The provider should reward the panelists for each study (not through raffles). Ask for the level of incentive paid for your study.
Risk:
Unserious and careless answers regardless of incentive.
Sometimes panelists will treat questionnaires with indifference even if the incentive is fair. They might go too fast through them, not reading the questions and answers carefully, not giving much thought to open-ended questions, contradicting themselves.
Solution:
The solution is to verify the database for several key indicators and to cancel the interviews that are problematic. For example, we cancel:
- Interviews that are completed too quickly (usually less than half of median time). Sometimes, if we have some questions that require more in-depth answers, we also measure the specific duration for that question, or we prevent the respondent from moving to the next question until a minimum amount of time has passed.
- Contradictory answers: either through answering in a straight line on a scale where we have antithetical statements; or contradicting something that they have answered earlier in the questionnaire.
- Trap options incorporated in the survey are checked by respondents: e.g. non-existent brands.
- Failed attention tasks throughout the questionnaire: e.g. asking them to check a specific answer option just to validate that they are paying attention and are not clicking randomly.
- Unserious answers to open-ended questions: gibberish, answers that do not make sense for the question.
Quality criterion in the RFQ phase:
Ask the provider what data quality validations they perform and ask them to include in the final database also the disqualified interviews and the reason for their disqualification.
Risk:
Low-quality data given by lack of transparency about the panels used and how they verify data quality.
If the provider is not transparent about the panels they plan to use and how those panels assure high-quality panelists and answers, the risk of low-quality data is high.
Solution:
We fully disclose from the beginning the panel sources, how the panelists are rewarded and how they are recruited. This way you have a clear picture of how each panel source is reducing the risks of low-quality data.
Quality criterion in the RFQ phase:
Ask for the panel sources, how the respondents are recruited and how they are rewarded.
In short, you can avoid these risks by asking for full transparency from the provider, requesting the following:
- How they make sure the respondents are real.
- How they make sure the respondents do not answer multiple times.
- How the respondents are recruited.
- What the incentive amount is, and if the respondents are rewarded for screen-outs and quota full.
- What data quality checks they conduct.
- If there is any pre-screening questionnaire.
- Which panels they work with and how those panels recruit and reward their panelists.
- Delivering the entire database at the end of the study, including screen-outs, quota-full, cancelled interviews, and the reasons for cancelling each interview.