Cognitive Interviewing

Overview

Cognitive interviewing is a popular research method used to evaluate survey questions. Federal statistical agencies, like the National Center for Health Statistics, frequently use cognitive interviewing to evaluate questions. The Collaborating Center for Question Design and Evaluation Research (CCQDER) uses cognitive interviewing more often than any other question evaluation method.

Cognitive interviewing provides a detailed picture of the meanings and processes respondents use to answer questions. These meanings and processes ultimately affect the data gathered through surveys. Cognitive interviewing offers insight that can improve understanding of question validity and response error. A question has validity if it measures what it's meant to measure. Response error happens when a respondent answers a question incorrectly.

Characteristics

A cognitive interview study has the following characteristics:

• Researchers select participants because they have desired qualities or experiences (purposive sample selection).
  • The goal is to identify problems, not make estimations or identify causes.
  • That's why a randomly selected sample is not required.
• Study samples are typically small, with 20 to 50 respondents.
• Researchers use appropriate methods to collect and analyze qualitative data.
  • Qualitative data are information, ideas, and observations that can't be shown as numbers.

Follow-up questions

Follow-up questions, or cognitive probes, are used to explore thought processes.

Probes can focus on specific thought processes:

• Comprehension: What do you think this term means?
• Recall: How did you remember that?
• Judgment: How sure are you about that?
• Response: How did you pick an answer to that?

Probes can also be more general and draw out information about the respondent's circumstances:

• Why did you answer that way?
• How did you come up with your answer?
• Tell me more about that.

Findings

Findings of a cognitive interview study:

• Are based on information that describes how respondents answered questions
• Are grounded in the respondents' experiences
• Provide insight into how respondents interpreted questions and considered their answers

Analysis

Analysis of qualitative interview data includes—

• Carrying out data collection and analysis at the same time using the 5 levels of analysis
• Developing codes and themes used for analysis based on the data (grounded theory)
• Making comparisons during each stage of analysis. For example:
  • Exploring question response in one interview (comprehension, recall, judgment, and response)
  • Comparing that finding to the next interview. Is it the same? If not, how is it different?

Levels of analysis

Level 1 – Conducting interviews
Level 2 – Summarizing interview notes
Level 3 – Comparing across respondents
Level 4 – Comparing across groups
Level 5 – Drawing conclusions about question performance