Key points
- Question evaluation is a crucial element in developing quality health surveys.
- Evaluation findings help improve survey data collection by increasing question validity and reducing response and non-response error.
- Quality question evaluation should be empirical, transparent, and systematic.
Benefits
Question evaluation is an important part of developing a good survey. Question evaluation also supports researchers who analyze data collected from surveys.
Evaluating survey questions reduces the risk that respondents will answer a question incorrectly because they don't understand the question (response error). It also reduces the risk that respondents won't answer a question at all (non-response error).
Reducing these risks also helps ensure that data can be compared across groups. If people in different groups understand questions differently, their responses won't mean the same things and shouldn't be compared.
Features
High-quality question evaluation should be empirical, transparent, and systematic.
Empirical
Empirical question evaluation is based on directly observed evidence, not based on expert opinion.
Transparent
Methods and analyses are clearly documented, so all findings and conclusions can be traced back to the original data. This transparency increases the credibility of the findings from question evaluation studies.
Systematic
Applying procedures consistently and strictly when collecting and analyzing data helps to ensure findings aren't inaccurate or misleading because of bias. Bias occurs when study procedures and practices make it more likely the study will produce some possible results and not others.
Q-Notes
Findings
Evaluation findings can tell us if our questions will produce accurate responses and get us the data we want. This is called question performance.
Ideally, questions should accurately present the concepts and topics that the survey researchers and designers want to study. These are the intended constructs. Knowing if questions are not presenting the intended constructs is important so questions can be changed before they are used.
A thorough study of question performance allows researchers to understand the potential range of constructs the data may represent. This is how question evaluation can enhance question validity. Validity is the extent to which a question measures what you wanted it to measure
Response patterns can differ across respondent groups. Respondents may interpret a question differently based on their personal experiences and characteristics. For example, respondents with different education levels might understand or interpret the terms used in a question differently.
Looking at response patterns across respondent groups can help researchers better understand and reduce subgroup bias. Subgroup bias occurs when people in different groups are more likely to interpret questions differently, while people in the same group are more likely to interpret them similarly. When this happens, data can't be compared across groups because the data may not mean the same thing from group to group.
There are many possible reasons why respondents might have difficulty answering a question. The question might be confusing, or the respondent might want to give the answer they think will make them look good (social desirability bias). Findings from question evaluation studies can reduce item non-response when findings are used to revise questions so that respondents are willing and able to answer.
Policy requirements
In Section 1.4 of The Standards and Guidelines for Statistical Surveys, the Office of Management and Budget (OMB) states:
"Agencies must ensure that all components of a survey function as intended when implemented in the full-scale survey and that measurement error is controlled by conducting a pretest of the survey components or by having successfully fielded the survey components on a previous occasion."
OMB oversees and coordinates the federal statistical system. The National Center for Health Statistics (NCHS) is part of the federal statistical system. CCQDER is an NCHS program.