Qualitative studies
Compared with quantitative methodologies (CATI, CAPI, CAWI), the risks in qualitative studies are lower, because you get to see the actual respondents and listen to them.
Still, this does not mean that you can choose any provider for a qualitative study. There are still risks that you must pay attention to if you want to obtain rich and meaningful findings about your topic of interest.
Risk:
Respondents do not match the required profile.
This is THE risk in qualitative research; all the others pale in comparison. If you do not have the right people, then you cannot rely on the information they provide.
The risk can have two sources:
- Respondents lying just to get the participation incentive and the provider not identifying this lie.
- Recruiters “training” respondents on how to answer in order to qualify.
Solution:
We recruit from two sources, which gives us access to a larger pool of potential participants:
- From our online panel which we also use for CAWI studies.
- The classic method of recruiting through a network of recruiters.
For recruiting through the online panel we use the same quality controls as for CAWI studies:
- A unique link for each respondent ID (no duplicate answers, no possibility to restart answering after screenout).
- Eliminating speeders, respondents who provide unserious answers to open-ended questions, respondents who give contradictory answers, those who check the trap options that we intentionally add.
- Adding questions and answers so that it is not clear for respondents what the subject of the study is and thus preventing them from knowing the profile we are looking for.
For the network of recruiters, the most important aspect is not telling the recruiters the full profile we are looking for. We provide only some general demographic characteristics (e.g. gender, age, city) and we keep all the other more specific characteristics (e.g. usage, attitudes, awareness) for a second stage of confirming respondents. This way, there is no opportunity for recruiters to instruct respondents on how to answer in order to qualify.
It is extremely important to have a second stage in the recruitment process where we validate the information provided by each respondent. We have a confirmation and validation team that:
- Calls, after a few days, the respondents from the online panel who completed the recruitment questionnaire and matched the required profile. The key recruitment questions are asked again by phone and, if their answers do not match those provided in the recruitment questionnaire, they are not invited to the study.
- Calls the respondents received from the field recruiters. They ask the key recruitment questions for the first time and those that do not match are not invited to the study.
- Verifies each respondent’s social media profile (Facebook, LinkedIn), to confirm key demographic information and the sector they work in.
We offer clients the full database of respondents, including those who do not match the required profile. For example, if you estimate an incidence of 20% for the required profile, those respondents should also represent around 20% of the full database.
Quality criterion in the RFQ phase:
Ask what the sources of recruitment are and how they plan to minimize the risk of respondents lying just to qualify and obtain the reward.
Ask for the full database of all recruits, including screen-outs. If the required profile appears miraculously without a corresponding proportion of screen-outs, there is likely a problem with the recruitment.
Risk:
Not having the required number of respondents for focus groups.
Even if respondents are confirmed several times before an interview or focus group, there will always be people who cancel at the last minute or simply fail to show up.
Solution:
Recruiting more people for backups. The required number of backup recruits varies from country to country. In Romania, from our experience, we usually need to recruit almost twice the required number of participants. For example, for a focus group with 6 participants, we usually recruit 12 people.
Quality criterion in the RFQ phase:
Ask the provider whether they recruit backup participants and how many they recommend for their market.
Risk:
Lack of information due to untalkative or unresourceful respondents.
The purpose of qualitative research is to explore your topics of interest as deeply as possible. If respondents are not very articulate or creative, they may not provide all the information you need.
Solution:
There is no foolproof solution to this risk, only methods to reduce it:
Add specific questions in the recruitment questionnaire: self-evaluation of their willingness to discuss different subjects; open-ended questions asking them to describe something (e.g. their last trip somewhere, their decision process for buying something).
Of course, these are not enough, since people sometimes overestimate themselves. That is why maybe the most important aspect is having someone with experience in recruitment that can have a conversation with that person during the recruitment process and evaluate their ability to take part in an in-depth interview or focus group.
Quality criterion in the RFQ phase:
Ask the provider how they assess whether respondents will actively contribute to the discussion.