Other Research Methods

About CCQDER additional research methods

Cognitive interviewing is a popular method for evaluating survey questions. In addition to cognitive interviewing, Collaborative Center for Questionnaire Design and Evaluation Research (CCQDER) researchers can also use several other methods to evaluate questions. These other methods may be used alongside of or instead of cognitive interviewing.

The goals and timeline of each project can help inform which other research methods are used. Some methods, like focus groups and ethnographic interviews, are used primarily during the questionnaire design phase. Other methods enhance cognitive interviewing during the evaluation phase.

Focus groups

Focus groups, or group discussions, are typically used during the question or questionnaire design phase of a project. These groups allow researchers to explore how potential respondents understand and talk about specific topics.

CCQDER researchers rely on the group dynamics and conversation that focus groups provide to get information about—

• Particular words or phrases respondents use to describe facts or events
• Which topics may be sensitive or burdensome

Focus groups are often combined with individual activities, such as short surveys or free-listing tasks. Individual activities can provide even more information about potential respondents' perceptions and understanding of the topic areas being studied.

Ethnographic interviews

Ethnographic interviews are one-on-one interviews that follow a flexible plan. These interviews are conducted during the question and questionnaire design phase of a project. Like focus groups, they allow CCQDER researchers to understand how potential respondents think and speak about a particular topic. This allows researchers to write questions about complex topics in ways that respondents understand.

The one-on-one nature of these interviews allows researchers to get a much deeper understanding about the way that potential respondents think and speak about a particular topic and how their position in society, personal characteristics and experiences affect how they understand the subject under investigation.

Usability testing

Usability testing involves observing how people interact with a questionnaire or other data collection instrument. When evaluating survey instruments that respondents will complete on their own, CCQDER often will combine usability testing with cognitive interviewing. Whether it's a pen and paper or web-based instrument, CCQDER researchers are interested in understanding how—

• Respondents interact with a questionnaire
• Their interactions affect their response processes

During usability testing, CCQDER researchers will typically observe these interactions and ask the respondent about them.

Web probing

Web probes are questions designed to draw out information about how respondents understand, think about, and respond to the questions that are being evaluated. Web probes encourage the respondent to write freely (open-ended) or require them to select from a list of responses (closed-ended). These probes are embedded in a web questionnaire after the question or questions being evaluated.

CCQDER is one of the world's leading research groups in developing and using web probes. While CCQDER has experience using and analyzing both open- and closed-ended probes, it focuses on close-ended probes. CCQDER uses closed-ended web probes developed directly from the findings of cognitive interviews.

Closed-ended web probes produce results that can be counted or measured (quantitative results). Quantitative results can be analyzed alongside the information, ideas, and observations from cognitive interviews (qualitative results). This allows for an in-depth, mixed-method approach that lets CCQDER researchers identify the distribution of interpretations and sources of measurement error. Researchers can make these identifications across a survey's intended population and within that population's subgroups.

This allows researchers to learn if certain groups are more likely than others to use out-of-scope interpretations beyond those intended by the researchers. Certain groups may include respondents with different education levels or from different areas of the country. Out-of-scope interpretations by certain groups could affect the accuracy of the data collected from the final survey.

CCQDER uses its semi-annual Research and Development Survey to evaluate questions using this innovative method.