How Do You Know Which Health Care Effectiveness Research You Can Trust? A Guide to Study Design for the Perplexed

Figure 1. Healthy user bias, a type of selection bias, is demonstrated in a study of 3,415 patients with pneumonia (and at high risk for flu and its complications), where elderly flu vaccine recipients were already healthier than nonrecipients. Figure is based on data extracted from Eurich et al (13).

Return to Article

The figure shows that the rate of flu-related death was 48% lower and the rate of hospitalizations for pneumonia or influenza was 27% lower among vaccinated elderly people than among unvaccinated elderly people.

Figure 2. A weak cohort study comparing the risk of death or hospitalization for pneumonia or flu among vaccinated versus unvaccinated elderly: example of failure to control for healthy users. Figure is based on data extracted from Nichol et al (15). 

Return to Article

Bar chart shows a reduced risk of death of 45% before flu season, of 39% during flu season, and of 26% after flu season.

Figure 3. Healthy user bias: a strong controlled study disproving the effects of the flu vaccine on all-cause mortality in the elderly during the flu “off season” (control period). The cohort study compared vaccinated elderly and unvaccinated elderly. Figure is based on data extracted from Campitelli et al (17).

Return to Article

Figure 4. Example of selection bias: underlying differences between groups of medical providers show how they are not comparable in studies designed to compare providers using EHRs with providers not using EHRs. Figure is based on data extracted from Simon et al (23) and Decker et al (24). Abbreviation: EHR, electronic health record.

Return to Article

Figure 5. Example of weak post-only cross-sectional study that did not control for selection bias: the study observed differences between practices with EHRs and practices with paper records after the introduction of EHRs but did not control for types of providers adopting EHRs. Note the unlikely outcome for nonsmoker. Figure is based on data extracted from Cebul et al (26). Abbreviations: BMI, body mass index; EHR, electronic health record.

Return to Article

Figure 6. Differences in patient characteristics between EHR-based practices and paper-based practices in a weak post-only cross-sectional study that did not control for selection bias. Abbreviation: EHR, electronic health record. Figure is based on data extracted from Cebul et al (26).

Return to Article

A diagram shows how randomization often ensures a fair comparison when assessing the effects of an intervention. The intervention begins with a population (eg, patients, health centers). A picture of hand flipping a coin illustrates how randomization often eliminates selection bias. The flip of the coin decides who is randomized to the intervention and who is not. Each arm of the study (intervention and no intervention) produces results, and the results are compared.

Figure 7. Randomized controlled trial: the “gold standard” of research design.

Return to Article

Figure 8. A strong randomized controlled trial of the effect of health information technology on the prevention of drug-related injuries among nursing home residents. Intervention participants received computerized warnings about unsafe combinations of drugs. Figure is based on data extracted from Gurwitz et al (30).

Return to Article

Figure 9. Elderly people who begin benzodiazepine therapy (recipients) are already sicker and more prone to fractures than nonrecipients. Figure is based on data extracted from Luijendijk et al (35).

Return to Article

Among previous users, the authors found 63 fractures and odds of a hip fracture of 0.9. Among current users, the authors found 73 fractures and odds of a hip fracture of 1.75.

Figure 10. Weak post-only epidemiological study suggesting that current users of benzodiazepines are more likely than previous users to have hip fractures. Figure is based on data extracted from Ray et al (32).

Return to Article

The figure presents 4 graphs. See text for complete description.

Figure 11. Several examples of effects that can be detected in interrupted time-series studies. The blue bar represents an intervention.

Return to Article

This figure consists of 2 graphs. The first shows a 60% decrease in benzodiazepine use in New York among female users after policy was implemented but no similar decrease in New Jersey. The second graph shows no change in the risk of hip fracture for either New York or New Jersey after policy was implemented.

Figure 12. Benzodiazepine (BZ) use and risk of hip fracture among women with Medicaid before and after regulatory surveillance restricting BZ use in New York State. A BZ user was defined as a person who had received at least 1 dispensed BZ in the year before the policy. From Annals of Internal Medicine, Wagner AK, Ross-Degnan D, Gurwitz JH, Zhang F, Gilden DB, Cosler L, et al. Effect of New York State regulatory action on benzodiazepine prescribing and hip fracture rates. 2007;146(2):96–103 (33). Reprinted with the permission of American College of Physicians, Inc.

Return to Article

Figure 13. Underreporting of calories and fat consumption due to social desirability among women and men. Figure is based on data extracted from Hebert et al (38). Fat intake was measured as the absolute percentage change for every 1% change in social desirability bias. The zero-line indicates no underreporting.

Return to Article

Figure 14. Study that contaminated intervention group by unwittingly tipping parents off to the socially desired outcome: fewer hours of television time per day for children. Figure is based on data extracted from Taveras et al (40).

Return to Article

The number of hours per week in television and computer use declined in 15 months by 17.5 hours in the intervention group and 3.4 hours in the control group. This difference in decline between the 2 groups was significant.

Figure 15. Strong randomized controlled trial design using an electronic device that caused an involuntary reduction in television and computer use. The difference in decline in viewing between the intervention group and control group was significant. Figure is based on data extracted from Epstein et al (42).

Return to Article

From preintervention to postintervention, the percentage of patients who received beta blockers increased from 60% to 78%, the percentage who received thrombolytics increased from 67% to 79%, and the percentage who received lidocaine decreased from 25% to 12%.

Figure 16. Percentage of acute myocardial infarction patients who received essential life-saving drugs (β blockers and thrombolytics) and a drug linked with increased mortality (lidocaine) in control hospitals before and after an intervention. Figure is based on data extracted from Soumerai et al (43).

Return to Article

Figure 17. Example of a weak post-only study of a hospital safety program and mortality that did not control for history. Narrow bar shows start of quality of care program. There is no evidence that data are available for the years leading up to the program. The study did not define the intervention period other than to state that planning occurred in 2003. Figure is based on data extracted from Pryor et al (45). Abbreviation: FY, fiscal year.

Return to Article

Figure 18. Example of a strong time-series design that controlled for history bias in the Institute for Healthcare Improvement’s 100,000 Lives Campaign. Figure is based on data from the Agency for Healthcare Research and Quality (48).

Return to Article