CAPI studies are probably the ones with the highest risk of fraud. Even if the data quality improved by switching from completing the interviews with pen and paper to tablets, the main problems remained.
When we were the first company to introduce audio recordings of F2F interviews in Romania many years ago, from a network of 300 interviewers we remained only with 17 of them who accepted this verification.
Another more recent example is in 2021 when we had to work with a new network of interviewers in a specific city in Romania and we had to invalidate 25% of the interviews due to quality reasons.
Thus, good CAPI studies are expensive.
Here are some risks you must pay attention to and how we avoid them:
Risk:
Uneven sample by not respecting the interview location or interview route.
Some interviewers do not follow the planned route for in-home interviews, or the planned fixed locations. This leads to an uneven sample that skews the data in unpredictable ways.
Solution:
Recording the GPS location for each interview and have in place solutions to detect if the interviewer tried to use fake GPS apps.
Quality criterion in the RFQ phase:
The supplier should be willing to provide the GPS coordinates for each interview, as well as details of the solution used to detect or prevent the use of fake GPS apps.
Risk:
There is no respondent, the answers are fictional.
Some interviewers might try to conduct or to finish interviews without an actual respondent. They might do this because it is easy money, because they get frustrated if the target incidence and the response rate are low, because the respondent drops out middle of the interview and they are disheartened by the wasted time, because they are pressured to finish the data collection quicker. Whoever did not try at least once to conduct a phone interview does not know how hard it might get sometimes.
Solution:
We audio record all interviews and compare the recordings with the database info.
We check that the same respondent is at the beginning of the questionnaire and at the end. We share all audio recordings with the client.
We also record the GPS location and start and finish hours of each interview, thus we validate that the interview took place in the designated spot or on the designated route, that interviews are not conducted at unusual hours or unusually fast one after another.
Quality criterion in the RFQ phase:
The supplier should record all interviews and be willing to send you all the recordings. GDPR should not be used as a banket justification for withholding recordings, as they represent a legitimate quality control practice.
The supplier should also provide the GPS location of each interview and the timing and duration of each interview.
Risk:
Respondents exist, but are not part of the target group.
This is more likely in low incidence studies where it is easier to “train” friends or acquaintances to act as respondents than finding the ones that actually qualify. Interviewers may call their friends or acquaintances and conduct the interviews by phone like they would have face-to-face. Or they might go to their friends or family and conduct several interviews in person.
Solution:
We record the GPS location of each interview to verify that interviewers adhere to the designated interview location for central-location and exit interviews, and follow the prescribed route for in-home interviews. To ensure data integrity, we also use Android Mock Location detection to prevent the use of fake GPS applications.
We review the timing of every interview and identify potentially suspicious interviews, including those conducted consecutively within unusually short intervals or at atypical hours. These checks help detect cases where respondents may have been prepared or recruited in advance.
Also, we provide audio recordings for all non-completed interviews (refusals, quota full, screen-outs) and the full database showing these non-completed interviews.
The number of screen-outs should be in the ballpark of what you anticipated. For example, if you have a target group with an estimated 20% incidence rate, then there should be 4 times more screened out interviews than completed ones.
Quality criterion in the RFQ phase:
The supplier should provide you also with audio recordings of all non-complete interviews, with the database showing all these non-complete interviews, with the GPS location of each interview and the start and finish hour of each interview.
Risk:
Untrained interviewers.
Even if the route was respected, the respondent exists and was correctly selected, an untrained interviewer may still ask questions the wrong way and consequently get irrelevant answers, or have a higher rate of refusals.
Solution:
Each interviewer goes through general training and specific training for each study. We ask all interviewers to conduct several test interviews for each study with our supervisors. The supervisors play the role of the respondent, play more complex scenarios to test how the interviewers manage them and give feedback live to interviewers on what to improve. We record both the general training and the test interviews for each study.
Quality criterion in the RFQ phase:
Ask to see the general interviewer training and the specific training for your study, as well as the proof that each interviewer working on your study went through the training successfully.
Risk:
Fraud or mistakes from interviewers.
Without effective verification, all quality criteria are empty words, and any kind of fraud may pass if nobody looks. There are almost unlimited situations through which fraud or mistakes can happen:
- not reading all questions,
- rephrasing questions and thus changing the meaning,
- not following the interviewer instructions (having to read or not the answer options, having to pause or not after each option and wait for a yes/no answer),
- not clarifying answers given by respondents – e.g. respondent says “satisfied” when the options are “Very satisfied” and “somewhat satisfied”,
- checking another answer than the one given by the respondent (either by mistake or willingly).
Solution:
We verify 10% of interviews conducted by each interviewer and compare them with the info registered in the database. If mistakes are found, all the interviews of that interviewer are verified and the feedback given quickly to the interviewer so that the mistakes are avoided for future interviews.
We correct in the database the mistakes that do not affect the rest of the data. If the mistakes affect the data, the interview is cancelled.
The 10% of interviews are not selected randomly, these are the interviews with the highest probability of having problems (e.g. shortest ones, unusual demographics). Therefore, it is more likely we will find most problems in those 10% of interviews, and not 10% of the problems.
Interviewers who do not make any mistake in a month receive an additional bonus of 10% for that month. Because a small lie in one single interview can wipe out 10% of the income in one month, the incentive to comply with the quality rules imposed by Mercury Research is high.
We can provide clients all the checking conducted, all the mistakes found and all the feedback given to interviewers.
Quality criterion in the RFQ phase:
Ask for the checking process: how many interviews are checked, how they are selected, what is the procedure when a mistake is found. Ask for a document with all the checks conducted, all the problems found, the decisions to accept or not those interviews or interviewers in the project, and the feedback given to interviewers.
Risk:
Lack of transparency regarding subcontractors and diluted responsibilities.
If the provider works with subcontractors for data collection, it might lead to unclear responsibilities for quality control: Does the provider verify the data quality or the subcontractor? If the subcontractor is checking, do you have access to the checks conducted?
Solution:
If we work with subcontractors we are transparent from the quoting stage about the name of the subcontractor. We also conduct all the data quality checks on the subcontractor’s interviews, applying the same standards as for our interviewers.
Quality criterion in the RFQ phase:
Ask for the name of the subcontractors, and for who is responsible for data checking.
In short, you can avoid these risks by asking for full transparency from the provider, delivering the following:
- Proof of training:
- Full audio recordings of all test interviews conducted by interviewers during the training period for each study.
- All the feedback given to interviewers for those test interviews.
- Proof of interviews conducted in the correct place/ on the correct route:
- GPS position of each interview.
- Proof of interview:
- Full audio recordings of all conversations made for the survey, whether completed or incomplete interviews (screen-outs, refusals, interrupted interviews).
- Full database including incomplete and disqualified interviews.
- Proof of verification:
- List of all verified interviews, the description of the verification, the non-conformities observed, the actions taken (validation, invalidation, correction).
- The emails with the feedback given to each interviewer.
- Proof of responsibilities:
If they work with subcontractors, and how the checking responsibility is divided.