CATI
Here are some risks you must pay attention to and how we avoid them:
Risk:
Sample does not represent the studied population.
Some suppliers have phone numbers from friends, acquaintances and previous studies, which are more likely (and thus cheaper) to participate in survey but do not represent well enough the population you investigate.
Solution:
We call on fully random generated numbers. For full transparency, we can show you how to generate the numbers in Excel and we will call only on those numbers received from you.
To demonstrate this, we will give you the call log from the phone provider at the end of the study that shows all the phone calls with time and duration, matching the info from the database.
Because this means calling on many inactive numbers, we also deploy a software that validates the active numbers and only those are directed to the interviewers.
Quality criterion in the RFQ phase:
The supplier should be willing to work with random generated number that is pre-selected by you and demonstrate that they called only on those numbers.
Risk:
There is no respondent, the answers are fictional.
Some interviewers might try to conduct or to finish interviews without an actual respondent. They might do this because it is easy money, because they get frustrated if the target incidence and the response rate are low, because the respondent drops out middle of the interview and they are disheartened by the wasted time, because they are pressured to finish the data collection quicker. Whoever did not try at least once to conduct a phone interview do not know how hard it might get sometimes.
Solution:
We audio record all interviews and compare the recordings with the database info.
We check that the same respondent is at the beginning of the questionnaire and at the end. We share all audio recordings with the client.
Quality criterion in the RFQ phase:
The supplier should record all interviews and be willing to send you all the recordings. GDPR should not be used as a banket justification for withholding recordings, as they represent a legitimate quality control practice.
Risk:
Respondents exist, but are not part of the target group.
This is more likely in low incidence studies where it is easier to “train” friends or acquaintances to act as respondents than finding the ones that actually qualify.
Solution:
By calling on numbers randomly generated by you and proving this with the call log, as described above.
Also, we provide audio recordings for all non-completed interviews (refusals, quota full, screen-outs) and the full database showing these non-completed interviews.
The number of screen-outs should be in the ballpark of what you anticipated. For example, if you have a target group with an estimated 20% incidence rate, then there should be 4 times more screened out interviews than completed ones:
Quality criterion in the RFQ phase:
The supplier should provide you also with audio recordings of all non-complete interviews and with the database showing all these non-complete interviews.
Risk:
Untrained interviewers.
Even if the sample is good, the respondent exists and was correctly selected, an untrained interviewer may still ask questions the wrong way and consequently get irrelevant answers, or have a higher rate of refusals.
Solution:
Each interviewer goes through general training and specific training for each study. We ask all interviewers to conduct several test interviews for each study with our supervisors. The supervisors play the role of the respondent, play more complex scenarios to test how the interviewers manage them and give feedback live to interviewers on what to improve. We record both the general training and the test interviews for each study.
Quality criterion in the RFQ phase:
Ask to see the general interviewer training and the specific training for your study, as well as the proof that each interviewer working on your study went through the training successfully.
Risk:
Fraud or mistakes from interviewers.
Without effective verification, all quality criteria are empty words, and any kind of fraud may pass if nobody looks. There are almost unlimited situations through which fraud or mistakes can happen:
- not reading all questions,
- rephrasing questions and thus changing the meaning,
- not following the interviewer instructions (having to read or not the answer options, having to pause or not after each option and wait for a yes/no answer),
- not clarifying answers given by respondents – e.g. respondent says “satisfied” when the options are “Very satisfied” and “somewhat satisfied”,
- checking another answer than the one given by the respondent (either by mistake or willingly).
Solution:
We verify 10% of interviews conducted by each interviewer and compare them with the info registered in the database. If mistakes are found, all the interviews of that interviewer are verified and the feedback given quickly to the interviewer so that the mistakes are avoided for future interviews.
We correct in the database the mistakes that do not affect the rest of the data. If the mistakes affect the data, the interview is cancelled.
The 10% of interviews are not selected randomly, these are the interviews with the highest probability of having problems (e.g. shortest ones, unusual demographics). Therefore, it is more likely we will find most problems in those 10% of interviews, and not 10% of the problems.
Interviewers who do not make any mistake in a month receive an additional bonus of 10% for that month. Because a small lie in one single interview can wipe out 10% of the income in one month, the incentive to comply with the quality rules imposed by Mercury Research is high.
We can provide clients all the checking conducted, all the mistakes found and all the feedback given to interviewers.
Quality criterion in the RFQ phase:
Ask for the checking process: how many interviews are checked, how they are selected, what is the procedure when a mistake is found. Ask for a document with all the checks conducted, all the problems found, the decisions to accept or not those interviews or interviewers in the project, and the feedback given to interviewers.
Risk:
Fast data collection leading to uneven sample.
This one is mainly on you: sometimes you want a very quick data collection, for example 1000 interviews collected in 1-2 days. Which is understandable, some business questions require immediate answers.
But what we have learned the hard way is that forcing the data collection will lead to uneven samples, reaching more women, and more inactive persons, because these categories are the ones that answer their phone more easily and have more time to participate to studies. It is better to wait a few more days for the results, but to know you can rely on them, that they include people who are busier and less likely to respond on the spot, than to push for a fast collection.
In short, you can avoid these risks by asking for full transparency from the provider, delivering the following:
- Proof of training:
- Full audio recordings of all test interviews conducted by interviewers during the training period for each study.
- All the feedback given to interviewers for those test interviews.
- Proof of calling on random generated numbers:
- Detailed phone bill with the duration and phone number for each call, with all the random numbers generated by you.
- Proof of interview:
- Full audio recordings of all conversations made for the survey, whether completed or incomplete interviews (screen-outs, refusals, interrupted interviews).
- Full database including incomplete and disqualified interviews.
- Proof of verification:
- List of all verified interviews, the description of the verification, the non-conformities observed, the actions taken (validation, invalidation, correction).
- The emails with the feedback given to each interviewer.