Evaluating Survey Questions in the Age of COVID-19

HOST: The quality of data in any health survey depends on the quality of the questions being asked, as well as the interpretation of those questions by the survey participants.  NCHS has a team in place that directly deals with those issues, the Collaborating Center for Question Design and Evaluation Research, or “CCQDER.”  CCQDER uses cognitive interviewing, a popular method for evaluating survey questions, by offering a detailed depiction of meanings and processes used by respondents to answer questions, which ultimately impact the survey data.  The sample of respondents in these studies is usually small, between 20 and 50 respondents.

This week, the CCQDER team hosted a webinar in which they discussed the design of survey questions about COVID-19, and the administration of those questions during the pandemic.  In addition to cognitive interviewing, the webinar also covered topics such as the Research and Development Survey, or “RANDS.”

As part of the Q and A segment of the webinar, Kristen Miller, the CCQDER Director, was asked whether standardized COVID questions had been developed by NCHS for outside researchers to use in their own studies:

KRISTEN MILLER: So traditionally it hasn’t been in our job scope to come up with standardized questions.  What we have done is – and maybe we want to rethink this for the future – but what we do do is we test these questions…  in these reports you will see specifically what each individual question captures. So what we would like to see people do is to get on, read these reports, see if this question – is this question capturing what I need it to capture – and then you making the decision, “Yes, I’m going to go with this question.”  So again, not anything standardized, but information provided to you so that you can choose the question that is best going to fit your research question.

HOST: The team was also asked what they would recommend in a rapid deployment situation in which there is not enough time to conduct a full evaluation of questions:

KRISTEN MILLER: I fully appreciate the problem.  And at the same time, I think that whenever we’re writing questions we need to have a concerted effort to have a plan how we’re going to go about question evaluation.  So it’s keeping track of the questions that go into the field, having mixed method or having these follow up pro questions that we had on RANDS to be able to see, “OK, this is going to be, there’s going to be error in this question, it’s going to be more error for less educated people, let’s keep that in mind as we interpret the data that’s coming in.”  But then, again, keeping track of what we’re asking so that we can improve our questions.  I mean, we’re so far into this pandemic I’d like to think our questions that we’re asking are much more improved from the questions that we began with when it first started.  So again, it’s just really having a question evaluation plan going forward.

HOST: The reaction to any survey question is highly personal and subjective, and the CCQDER team was asked about whether respondents have been impacted by their fear of COVID-19 when answering the questions.  Dr. Stephanie Willson of the CCQDER team described some of the challenges:

STEPHANIE WILLSON: Right, actually that’s a very astute observation because again the experience that people had – you had people who thought it was a hoax over here to people who were super-afraid of getting it, right?  So that absolutely was filtered through.  One example I didn’t get a chance to talk about was this need — there were questions about, “Did you need medical care for something but not get it because of the pandemic?”  So that kind of fear, the idea of need, was filtered through people’s experience with the pandemic and how afraid they were of catching it.  So certain things were missed, because suddenly now, “I don’t need to get a check-up, I don’t need a well-woman visit, I don’t need a cancer screening because of the pandemic.”  Where in non-pandemic times:  “Yes, I feel like I need those things.”  So that is an element of fear that absolutely did factor into interpretations.

HOST: Dr. Willson also discussed the differences between “remote interviewing” and traditional face-to-face interviews, and whether remote interviewing will continue into the post-pandemic era:

STEPHANIE WILLSON: The interesting thing was, even though I’ve been doing this for a long time, I had never done a virtual interview prior to the pandemic so I went into it kind of skeptical.  But I have to tell you, I’m a convert.  I really feel as though Zoom interviews really gave the same kind of quality cognitive interview data that face-to-face, in-person face-to-face interviews gave, so I think we should continue to use this.  I think that in certain situations, there’s a downside maybe in terms of socioeconomic status, but the upside to this would be geographical diversity that we can now explore that you can’t do… It takes so much more money to, let’s say, go regionally throughout the United States, for example.  And we did have actually geographic diversity in our sample here – not enough to make it count because we were trying to do it quickly but, yeah, I think it should continue to be used.

HOST: All CCQDER studies feature a final report that document the study findings, and are housed on a searchable, publicly accessible database called Q-Bank.

MUSICAL BRIDGE

HOST: Urgent care centers and health clinics located within grocery or retail stores are able to provide acute health care services for non-emergency visits, and they also can provide preventive care services, such as routine vaccinations. The availability and utilization of urgent care has risen dramatically in recent years.

A new report released this week examines urgent care center and retail health clinic visits among adults in the past year by sex and selected characteristics.  The report uses data from the 2019 National Health Interview Survey, and reveal that 1/3 of women and slightly over ¼ of men made one or more visit to an urgent care center or retail health clinic in the past year.

Older adults are less likely to use urgent care centers or retail health clinics than younger adults, and non-Hispanic white adults are more likely to have visited an urgent care center or retail health clinic at least once in the past year compared to Hispanic, non-Hispanic black or NH Asian adults.  In addition, adults with higher education levels are more likely to use urgent care centers or retail health clinics.

A second report came out this week which compares provisional or preliminary 2020 data with final 2019 and 2018 data on changes in the number of births in the United States by month and by state.  The report also includes data on the race and Hispanic origin of the mother, and sheds some light on the impact of the COVID-19 pandemic on fertility in the country.

From 2019 to 2020, the number of births declined for each month.  In comparison, from 2018 to 2019, the number of births declined for only 9 months of the year.  The largest declines in 2020 occurred in December, followed by August, and then October and November.  The number of births in the U.S. declined 8% more in December 2020 than it did the previous year.

In other words, the decline in births between 2019 and 2020 was larger in the second half of the year than in the first half of the year – 6% in the second half of the year vs. only 2% in the first half.   Between 2018 and 2019, the number declined 2% in the first half of the year and 1% in the second half.

Ultimately, more information on fertility during the pandemic won’t be known until 2021 data are available. The first provisional data for 2021 should be available by early Fall.