The way relationships are important for understanding women empowerment was explored in the previous blog. This blog explores a method used to probe more deeply on comprehension of survey questions and to gather information on perceptions.
Cognitive interviewing is an evidence-based, qualitative method to assess participants’ understanding of survey questions before the survey is administered. Cognitive interviews can be conducted as part of the pre-testing phase prior to fieldwork. Cognitive interviewing requires the following:
ü Being an attentive observer. Recording of non-verbal cues, of verbal indicators of confusion or hesitancy, and information on the immediate environment related to the interview.
ü Other techniques. Such as asking the respondent to “think aloud” while formulating a response to a survey question; or the use of probing questions which may be asked immediately following the relevant question, retrospectively following the interview, or a combination of the two.
ü Being able to conduct a thematic analysis. Using a codebook to identify emerging themes directly from the data.
Regarding sample size, common practice ranges from 5 to 15 interviews per language group. The goal is not to obtain sample sizes large enough to supply precision in statistical estimates, but rather to maximize variance among the respondent group by interviewing a variety of individuals who would be useful in informing decisions about if and how to modify questions.
Why cognitively test?
Even when questions are found to be generally understood, participants may occasionally respond to the wrong part of the question, they may have not understood key phrases, or feel uncomfortable answering the questions. Cognitive interviewing is therefore a useful approach not only to understand whether questions are understood as intended, but also to capture the motives for responses, and whether the given responses actually reflect participants’ experiences. The examples below offer some illustrations of when there has been a disconnect between the question asked and how it was answered. The examples comes from Bangladesh:
- Respondents understand “contraceptive method” as “not having a child”; however, the original question had asked about specific methods.
- Respondents understand “special foods for children (i.e. foods specifically designated for children and not consumed by adult HH members)” as a variety of food items, including eggs, milk, fruits, and vegetables, despite special foods for children (e.g., infant cereals) being common in the area.
- Respondents interpret the term “respect” in the intrahousehold relationships module as “honour”.
- Certain women find it problematic to address this question “comfort in telling husband if you disagree with him”.
Based on cognitive interviewing, solutions may be developed by: (a) modifying the wording of questions; (b) providing more specific instructions and specifically offering culturally appropriate examples; (c) defining terms explicitly. Qualitative insights originating from cognitive interviewing can provide new insights into survey questions and suggest the need for new (more culturally and linguistically relevant) questions.
Sometimes, researchers may not realise that the respondents are answering a different question to what they asked until the end of the data collection process, when it is too late to correct. For example, researchers asked the following question:
“Do you feel comfortable speaking up in public about any issue that is important to you, your family, or your community?”
In Uganda the word “issue” translates to “problem” or “challenge” in local languages and thus has a more negative rather than neutral connotation. Women empowerment, therefore, in order for people to include issues with both positive and negative connotations, any issue was changed to read anything. With this small change in wording, a higher percentage of respondents gave a neutral definition of “issue” in the subsequent round of testing. Despite the improvement in the wording of this question, there were too many concerns about the social or cultural acceptability of the question to keep using it.
Cognitive interviewing is emerging as an effective approach that should be a routine part of questionnaire design and testing. Even after cognitive testing has finished, it is a good idea to further refine and test the tools before enumerators independently begin asking questions.
Our next blog explores ways to capture social, gender norms and women empowerment.
[1] Sources: Hannan, A., Heckert, J., James-Hawkins, L., & K.M. Yount (2019); Malapit, H.J.L., Sproule, K., & C. Kovarik (2017).
[1] R1 WEAI 2.0 [G6.01]
