About
This ACIP GRADE handbook provides guidance to the ACIP workgroups on how to use the GRADE approach for assessing the certainty of evidence.
Summary
After assessing study limitations, inconsistency, indirectness, imprecision and publication bias, three criteria should be considered that may warrant raising the evidence level in nonrandomized studies (NRS): strength of association, dose-response gradient, and opposing plausible residual confounding or bias1.
9.1 Strength of association
When the strength of the association is strong and the effect of the estimate is large or very large, the GRADE assessment may be upgraded due to more certainty in the results1. If a study has no major concerns with confounding or internal validity, then it may be appropriate to upgrade the evidence level. If the effect is large enough, the observed benefit cannot be explained by weak study design alone and instead allows for consideration that there is some confidence in the estimate of the effect. Therefore, while NRS are likely to provide an overestimate of the true effect, a strong association in the effect size may lead to stronger certainty in the evidence. The evidence level may be upgraded by one level if the relative risk from at least two studies is approximately >2 or <0.5, and it may be upgraded by two levels if the relative risk is approximately >5 or <0.2. Table 7a shows a scenario in which a NRS was upgraded due to the strong association seen in the effect size.
Table 14. Relationship between effect measure and evidence level
| Strength of Association | Effect Measurea | Evidence Level |
|---|---|---|
| Strong | Relative Risk approximately >2 or <0.5 (based on consistent evidence from at least 2 studies) | Move up 1 level |
| Very strong | Relative Risk approximately >5 or <0.2 | Move up 2 levels |
a Relative risks of 0.5 and 0.2 correspond to vaccine efficacies of 50% and 80%, respectively. Vaccine efficacy = (1 – Relative Risk) x 100.
Table 15. Evidence profile for outcome of development of Ebola-related symptomatic illness
References in this table: 2
| Certainty assessment | No. of patients | Effect | Certainty | Importance | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No. of studies | Study Design | Risk of Bias | Inconsistency | Indirectness | Imprecision | Other considerations | rVSV-vaccine | No rVSV-vaccine | Relative (95% CI) | Absolute (95% CI) | ||
| 1 | Randomizeda (clusters) | Not serious | Not serious | Seriousb | Seriousc | None | 0/51 (0.0%) | 7/47 (14.9%) | RR 0.06d (0.94 to 6.91) | 140 fewer per 1,000 (from 149 fewer to 7 more) | Low Evidence | Critical |
| 1 | Non-randomized (particip ants) | Not serious | Not serious | Seriousb | Seriousc | Strong association | 0/210 8f (0.0%) | 16/30 75(0.5%) | RR 0.04d (0 to 0.74) | 5 fewer per 1,000 (from 5 fewer to 1 fewer) | Moderate Evidence | Critical |
CI: Confidence interval; RR: Relative risk; Note: Outcome assessed with laboratory confirmed case of EVD
Explanations
a. Henao-Restrepo 2017 was a cluster randomized trial (i.e., units of randomization were clusters); cluster-level data presented here.
b. Concern for indirectness to the U.S. population: population consists of contacts, and contacts of contacts of EVD cases, and ring vaccination strategy which may include post-exposure vaccination.
c. Because this study was done at a time when the 2014-2015 West Africa outbreak was waning in Guinea and there are few events reported, it does not meet optimal information size and suggests fragility in the estimate; 95% CI contains the potential for desirable as well as undesirable effects.
d. Henao-Restrepo 2017 was a cluster randomized trial (i.e., units of randomization were clusters); participant level data presented here.
e. The concerns with indirectness pose no inflationary effect; therefore, the evidence was rated up based on a very large magnitude of effect from the 96% reduction in risk and overall certainty was upgraded two levels.
f. RR calculated using the standard continuity correction of 0.5.
g. Denominator represents participants from the clusters randomized to receive immediate vaccination.
h. RR calculated using the standard continuity correction of 0.5.
9.2 Dose-response gradient
A dose-response gradient could upgrade the certainty of evidence assessment for NRS 1. For example, if greater vaccine efficacy corresponds with increasing number of doses in a series, then the dose response relationship may result in more confidence in the results. While residual confounding could contribute to the effect estimate, if the effect size is large and a dose-response gradient is observed, it is likely that confounding alone cannot account for the strength of the association; therefore, the evidence level may be upgraded.
9.3 Opposing plausible residual confounding or bias
Both randomized control trials (RCTs) and NRSs may be impacted by plausible bias that underestimates the effect of an intervention or increases the effect when no effect was observed1. For example, if a vaccine is suspected of being associated with an adverse event and the publicity results in increased spontaneous reporting of the adverse event among vaccinated persons compared to that in unvaccinated persons, yet epidemiological studies find no association, the evidence level for the lack of association can be upgraded. Similarly, if an intervention is given to sicker patients and the results still show that they improved more than the control group, the actual effect is likely larger than the observed effect estimate.
- Schünemann H, Higgins J, Vist G, et al. Chapter 14: Completing ‘Summary of findings’ tables and grading the certainty of the evidence. In: Higgins J, Thomas J, Chandler J, et al, eds. Cochrane Handbook for Systematic Reviews of Interventions version 63 (updated February 2022). 2022. www.training.cochrane.org/handbook.
- Choi MJ, Cossaboom CM, Whitesell AN, et al. Use of ebola vaccine: recommendations of the Advisory Committee on Immunization Practices, United States, 2020. MMWR Recommendations and Reports. 2021;70(1):1.