Grading of Recommendations, Assessment, Development, and Evaluation (GRADE): Moderna COVID-19 Vaccine

Overview

A Grading of Recommendations, Assessment, Development and Evaluation (GRADE) review of the evidence for benefits and harms for Moderna coronavirus disease 2019 (COVID-19) vaccine was presented to the Advisory Committee for Immunization Practices (ACIP) on December 19, 2020. GRADE evidence type indicates the certainty in estimates from the available body of evidence. Evidence certainty ranges from type 1 (high certainty) to type 4 (very low certainty)1.

The policy question was, “Should vaccination with Moderna COVID-19 vaccine be recommended for persons 18 years of age and older during an Emergency Use Authorization?” The potential benefits pre-specified by the ACIP COVID-19 Vaccines Work Group included prevention of symptomatic laboratory-confirmed COVID-19 (critical), hospitalization due to COVID-19 (critical), all-cause death (important), SARS-CoV-2 seroconversion (important), and asymptomatic SARS-CoV-2 infection (important). The two pre-specified harms were serious adverse events (critical) and reactogenicity grade ≥3 (important).

A systematic review of evidence on the efficacy and safety of a two-dose regimen of Moderna COVID-19 vaccine among persons aged 18 years and older was conducted. The quality of evidence from two Phase I dose-ranging studies, one Phase II randomized controlled trial, and one Phase III randomized controlled trial were assessed using a modified GRADE approach23456.

A lower risk of symptomatic COVID-19 was observed with vaccination compared to placebo (relative risk [RR] 0.06, 95% confidence interval [CI]: 0.03, 0.11, evidence type 1), corresponding to a vaccine efficacy of 94.1% (95% CI: 89.3%, 96.8%). This was observed with a median follow-up of two months, prompting concern for indirectness due to the short duration of follow-up (i.e., observed outcome of vaccine efficacy at two months does not directly inform vaccine efficacy for any duration longer than two months). The Phase III randomized controlled trial that generated this estimate is ongoing, thus the efficacy estimate may change with additional follow-up. However, given the strength of the association (i.e., high estimated vaccine efficacy), it is unlikely that the efficacy estimate for prevention of symptomatic COVID-19 would change substantially enough in the months following vaccination to fall below the FDA-defined efficacy threshold for an Emergency Use Authorization. The vaccine was also associated with a lower risk of hospitalization due to COVID-19 (RR 0.11; 95% CI: 0.01, 0.87; evidence type 2, serious concern for indirectness), corresponding to a vaccine efficacy of 89% (95% CI: 13%, 99%). In addition, the vaccine was associated with a lower risk of all-cause death (RR 0.86; 95% CI: 0.29, 2.55; evidence type 4, serious concern for indirectness, very serious concern for imprecision). Preliminary data were available regarding prevention of asymptomatic SARS-CoV-2 infection; SARS-CoV-2 PCR test results from nasopharyngeal swabs collected on the day of the second vaccine dose indicated lower risk of asymptomatic infection among vaccine recipients compared to placebo recipients (RR 0.37; 95% CI: 0.20, 0.68; evidence type 4, serious risk of bias and very serious concern for indirectness).

In terms of harms, the available data indicated that serious adverse events were balanced between the vaccine and placebo arms (RR 0.96; 95% CI: 0.77, 1.20; evidence type 2, serious concern for indirectness), and FDA judged three serious adverse events likely to be related to vaccination among over 15,000 persons vaccinated. Reactogenicity grade ≥3 was associated with vaccination (RR 4.93; 95% CI: 4.55, 5.34; evidence type 1). About 22% of vaccine recipients and 4% of placebo recipients reported any grade ≥3 local or systemic reactions following either dose 1 or dose 2.

The Phase III randomized controlled trial accrued the number of cases needed to meet the primary endpoint of vaccine efficacy against symptomatic COVID-19 and the secondary endpoint of vaccine efficacy against severe COVID-19, based on a protocol definition that did not require hospitalization. Hospitalizations due to COVID-19 and deaths are less common, thus, Phase III trials may not be designed or statistically powered to evaluate differences between vaccine and placebo arms for these outcomes. However, for hospitalization due to COVID-19, a statistical difference was observed based on only 10 cases in the Phase III trial. Since robust direct evidence is not expected from early results from Phase III studies, vaccine efficacy in preventing hospitalizations due to COVID-19 and deaths may also be inferred from observed efficacy against symptomatic COVID-19. Preliminary data from one study suggested possible short-term efficacy against asymptomatic SARS-CoV-2 infection after one vaccine dose, but no data were available to assess long-term efficacy or efficacy after two-dose vaccination series completion. No data were available for assessment of SARS-CoV-2 seroconversion. Data on SARS-CoV-2 seroconversion will be available from an ongoing Phase III trial.

Introduction

As of December 12, 2020, one vaccine had been recommended for prevention of COVID-19, caused by the SARS-CoV-2 virus which emerged in late 20197.

On December 19, 2020, the U.S. Food and Drug Administration (FDA) issued an Emergency Use Authorization (EUA) for Moderna COVID-19 (mRNA-1273) vaccine for prevention of symptomatic COVID-19 for persons aged ≥18 years8. As part of the process employed by the ACIP, a systematic review and GRADE evaluation of the evidence for Moderna COVID-19 vaccine was conducted and presented to ACIP. There were no conflicts of interest reported by CDC and ACIP COVID-19 Vaccines Work Group members involved in the GRADE analysis.

The ACIP adopted a modified GRADE approach in 2010 as the framework for evaluating the scientific evidence that informs recommendations for vaccine use. GRADE was used to evaluate the Moderna COVID-19 vaccine. Evidence of benefits and harms were reviewed based on the GRADE approach1.

The primary policy question was, “Should vaccination with Moderna COVID-19 vaccine be recommended for persons 18 years of age and older during an Emergency Use Authorization?” (Table 1).

Methods

We conducted a systematic review of evidence on the efficacy and safety of a two-dose regimen of Moderna COVID-19 vaccine. We assessed outcomes and evaluated the quality of evidence using the GRADE approach.

We identified studies in Medline, Embase, and Cochrane Library, written in English, without date restrictions. Search terms included coronavirus, COVID-19, SARS-CoV-2, respiratory (symptom, disease, illness, condition), vaccine, immunization, trial, double blind, single blind, placebo, comparative study, phase I, phase II, phase III, immunogenicity, efficacy, effective, adverse, evidence, and variations on these terms (see Appendix 2 for details).

Articles were included if they provided data on vaccination with the Moderna COVID-19 vaccine and 1) involved human subjects; 2) reported primary data; 3) included adults (ages 18 and older) at risk for SARS-CoV-2 infection; 4) included data relevant to the efficacy and safety outcomes being measured; and 5) included data for the specific vaccine formulation, dosage, and timing being recommended (mRNA-1273, 100 μg IM, 2 doses at 0 and 28 days). In addition, efforts were made to obtain unpublished and other relevant data by hand-searching reference lists, and consulting with vaccine manufacturers and subject matter experts. Titles and abstracts were screened independently and in duplicate by two separate reviewers. Characteristics of the included studies are shown in Appendix 1.

Patient-important outcomes (including benefits and harms) for assessment were selected by the Work Group during Work Group calls and via online surveys where members were asked to rate and rank the importance of relevant outcomes. The GRADE assessment across the body of evidence for each outcome was presented in an evidence profile.

Results

The results of the GRADE assessment were presented to ACIP on December 19, 2020.

After title and abstract screening of 2,824 records, 19 studies were identified as eligible for full-text review. Of these, 15 were excluded because they assessed a different vaccine. This left 4 studies for the evidence synthesis and GRADE evidence assessment23456. Characteristics of the included studies are shown in Appendix 1.

Outcomes of interest included individual benefits and harms. Benefits of interest deemed critical were prevention of symptomatic laboratory-confirmed COVID-19 and prevention of hospitalization due to COVID-19 (Table 2). Other important outcomes included prevention of all-cause death; SARS-CoV-2 seroconversion to a non-spike protein; and asymptomatic SARS-CoV-2 infection. The critical harm of interest was serious adverse events, including vaccine-associated enhanced disease; reactogenicity grade ≥3 was deemed an important harm. SARS-CoV-2 seroconversion was not included in the evidence profile because no data were available.

Four studies were reviewed that provided data on outcomes specified for GRADE (Appendix 1). Data were reviewed from two published Phase I studies, with additional data provided by the sponsor23. Data were reviewed from one Phase II randomized controlled trial and one Phase III randomized controlled trial using data provided by the sponsor and FDA456910. The final GRADE assessment was limited to the Phase II and III randomized control trial data.

The Moderna COVID-19 vaccine reduced symptomatic laboratory-confirmed COVID-19 when compared to no COVID-19 vaccination (vaccine efficacy: 94.1%; 95% CI: 89.3%, 96.8%) (Table 3a). For hospitalization due to COVID-19, 10 events were documented, 9 in the placebo group and 1 in the vaccine group. Vaccine efficacy against hospitalization due to COVID-19 was 89% (95% CI: 13%, 99%) (Table 3b). All-cause deaths were also uncommon, 6 in the vaccine group and 7 in the placebo group (RR: 0.86; 95% CI: 0.29, 2.55) (Table 3c). Vaccination with one dose was associated with a lower risk of asymptomatic SARS-CoV-2 infection, defined in a post-hoc analysis as SARS-CoV-2 DNA detection by polymerase chain reaction (PCR) testing on the day dose 2 was received and no COVID-19 symptoms reported between dose 1 and dose 2 (RR: 0.37; 95% CI: 0.20, 0.68), among persons who were SARS-CoV-2 seronegative at baseline (Table 3d).

For evaluation of potential harms, data were pooled from the Phase II and Phase III randomized controlled trials. Numbers of serious adverse events were comparable between the vaccine group and the placebo group across the two studies (RR: 0.96; 95% CI: 0.77, 1.20); there were no cases of vaccine-associated enhanced disease or vaccine-related deaths (Table 3e). Grade ≥3, or severe, local or systemic reactions 7 days following either vaccination, were reported by 22% of vaccine recipients, and occurred more frequently in the vaccine than placebo groups (Table 3f).

GRADE Summary

The initial GRADE evidence level was type 1 (high) for each outcome because the body of evidence was from randomized controlled trials (Table 4). In terms of benefits, the available data indicated that the vaccine was effective for preventing symptomatic COVID-19, and no serious concerns impacting certainty in the estimate were identified in the context of the time frame of an Emergency Use Authorization for this outcome (type 1, high). The certainty in the effect estimate for hospitalization due to COVID-19 was downgraded one point for serious concern of indirectness related to the median two months follow-up and the possibility that hospitalizations among persons meeting the criteria for severe COVID-19 may not have included all hospitalized COVID-19 cases (type 2, moderate). The certainty in the effect estimate for all-cause death was downgraded one point for serious concern of indirectness related to the median two months follow-up and two points for very serious concern of imprecision (type 4, very low). The certainty in the estimate of effect for asymptomatic SARS-CoV-2 infection was downgraded one point due to serious risk of bias and two points for very serious concern for indirectness. The certainty in the estimate of the effect for serious adverse events was downgraded one point due to serious concern of indirectness related to the median two months follow-up and sample size (type 2, moderate). No serious concerns impacted the certainty in the estimate of reactogenicity (type 1, high) (Table 4).

Table 1: Policy Question and PICO

Abbreviations: IM = intramuscular.

^a Assessed through serial PCR testing.

Table 2: Outcomes and Rankings

^a Limited number of events were observed in studies identified in the review of evidence.

^b Data on outcome not available in studies identified in the review of evidence.

Table 3a: Summary of Studies Reporting Symptomatic Laboratory-confirmed COVID-195

Abbreviations: RT-PCR = real-time polymerase chain reaction; CI = confidence interval; HR = hazard ratio.

^a Based on data cutoff November 25, 2020; participants had a median of two months follow-up;15208 and 15210 persons were randomized to vaccine and placebo, respectively.

^b Primary outcome, defined as SARS-CoV-2 RT-PCR-positive symptomatic illness, in seronegative adults, ≥14 days post second dose.

^c Symptomatic illness defined as at least one nasopharyngeal swab, nasal swab, or saliva sample (or respiratory sample, if hospitalized) positive for SARS-CoV-2 by RT-PCR; and at least two symptoms: fever (≥ 38ºC), chills, myalgia, headache, sore throat, new olfactory and taste disorder; or at least one symptom: cough, shortness of breath or difficulty breathing, or clinical or radiographical evidence of pneumonia.

^d Participants were considered to be at risk for severe COVID-19 if they had at least 1 of the following risk factors at screening: chronic lung disease, significant cardiac disease, body mass index ≥ 40 kg/m2, diabetes, liver disease, or controlled HIV infection.

Table 3b: Summary of Studies Reporting Hospitalization due to COVID-19910

Abbreviations: CI = confidence interval; RR = relative risk

^a Participants aged ≥18 years, without evidence of prior infection, ≥14 days after dose 2. Hospitalized cases are a subset of all severe COVID-19 cases meeting a protocol-specified definition of severe COVID-19.

Based on data cutoff November 25, 2020. There was one severe COVID-19 case in a vaccine recipient which occurred two months after dose 2, requiring hospitalization, that was not adjudicated by the data cutoff date.

Table 3c: Summary of Studies Reporting All-cause Death^a5

Abbreviations: RR = relative risk; CI = confidence interval.

^a Death from any cause, including COVID-related or serious adverse event, based on data collected through December 3, 2020, presented in FDA briefing document.

^b Estimate and confidence interval were calculated based on number of participants.

Table 3d: Summary of Studies Reporting Asymptomatic SARS-CoV-2 Infection6

Abbreviations: RR = relative risk; CI = confidence interval.

^a Data are presented from a post-hoc analysis among persons who were SARS-CoV-2 seronegative at baseline. Asymptomatic infection was defined as laboratory-confirmed SARS-CoV-2 infection from PCR testing on the day of the second vaccine dose and no reported COVID-19 symptoms between dose 1 and dose 2. Data on symptoms after dose 2 were not available.

Table 3e: Summary of Studies Reporting Serious Adverse Events^a235

Abbreviations: RR = relative risk; CI = confidence interval; RCT = randomized controlled trial.

^a Death, life-threatening event, hospitalization, incapacity to perform normal life functions, medically important event, or congenital anomaly/birth defect

^b Risk of bias was not formally assessed for these small studies with no comparator; these were not included in the quantitative estimate used for GRADE.

^c Seven serious adverse events were deemed by blinded investigators to be related to vaccination. These included: intractable nausea/vomiting; facial swelling (two reports); rheumatoid arthritis; dyspnea with exertion and peripheral edema; autonomic dysfunction; and B-cell lymphocytic lymphoma. Through further investigation by the FDA, only three were classified as related to vaccination: one report of intractable nausea/vomiting and two reports of facial swelling. The FDA concluded that the possibility that the vaccine contributed could not be excluded for rheumatoid arthritis, dyspnea and peripheral edema, and autonomic dysfunction. The FDA concluded that B-cell lymphocytic lymphoma was not related to vaccination.

Table 3f: Summary of Studies Reporting Reactogenicity^a234

Abbreviations: RR = relative risk; CI = confidence interval.

^a Reactogenicity outcome includes local and systemic events, grade ≥3. Grade 3: prevents daily routine activity or requires use of a pain reliever. Grade 4: requires emergency room visit or hospitalization. Grade 4 systemic adverse reactions were reported by 24 participants, 17 in vaccine group and 7 in placebo group.

^b Based on interim analysis, data cutoff November 11, 2020.

^c Risk of bias was not formally assessed for these small studies with no comparator; these were not included in the quantitative estimate used for GRADE.

Table 4. Grade Summary of Findings Table

Abbreviations: CI = confidence interval; RR = relative risk; COVID-19 = coronavirus disease 2019; RCT = randomized controlled trial.

^a Risk of bias related to blinding of participants and personnel was present. Although participants and study staff were blinded to intervention assignments, they may have inferred receipt of vaccine or placebo based on reactogenicity. This was deemed unlikely to overestimate efficacy or underestimate risk of serious adverse events, therefore the risk of bias was rated as not serious.

^b Concern for indirectness was noted due to the short duration of observation in the available body of evidence. The vaccine efficacy observed at a median two-month follow-up may differ from the efficacy observed with ongoing follow-up. However, in consideration of the strength of association and precision observed, it is unlikely that the efficacy estimate for symptomatic COVID-19 would change substantially enough to fall below the FDA-defined efficacy threshold for an Emergency Use Authorization (i.e. to <50% efficacy).

^c The effects noted are from a per protocol analysis with outcomes assessed at least 14 days post dose 2 among persons who received two doses and had no evidence of prior SARS-CoV-2 infection. In an interim analysis using the full analysis set (persons who received at least one dose, with or without evidence of prior SARS-CoV-2 infection), there were 21 cases among 15180 persons in the vaccine arm and 173 cases among 15170 persons in the placebo arm (RR = 0.12 (0.07 to 0.19)).

^d The RCT excluded persons with prior COVID-19 diagnosis, pregnant or breastfeeding women, and persons who were in an immunosuppressive or immunodeficient state, had asplenia or recurrent severe infections (HIV-positive participants on stable antiretroviral therapy were not excluded). The population included in the RCT may not represent all persons aged ≥18 years.

^e Absolute risk was calculated using the observed outcomes in the placebo arm during the available clinical trial follow-up. Absolute risk estimates should be interpreted in this context.

^f The effects noted are from a per protocol analysis with outcomes assessed at least 14 days post dose 2, among persons who received 2 doses, and had no evidence of prior SARS-CoV-2 infection.

^g Serious concern for indirectness was noted due to the short duration of follow-up in the available body of evidence. Severe COVID-19 cases leading to hospitalization may not have had time to occur in a median two-month follow-up. Additionally, hospitalization was ascertained on a subset of cases meeting a protocol-specified definition of severe COVID-19; hospitalization due to COVID-19 was not ascertained for all COVID-19 cases, and it is possible that hospitalizations for COVID-19 occurred in cases not meeting the specific severe COVID-19 criteria.

^h Includes one hospitalized case in the vaccine arm that had not yet been adjudicated by the data cutoff date.

ⁱ Serious concern for indirectness was noted due to the short duration of follow-up in the available body of evidence. The vaccine efficacy over a practical time frame for a vaccination program may differ from the short-term efficacy observed in the clinical trial. Deaths due to COVID-19 may not have had time to occur through December 3, 2020.

^j Imprecision assessed based on the width of the 95% confidence interval. This outcome may be imprecise due to small number of events reported during the observation period.

^k Death from all causes was considered a descriptive outcome in the clinical trial data. The sponsor provided counts of total deaths but appropriate denominators for analysis to evaluate benefits for this outcome are not clear. While the number of events is accurate, lack of an agreed upon denominator may introduce some fragility in the estimate.

^l Serious concern for risk of bias due to selective outcome reporting was present. Evaluation of asymptomatic infection from SARS-CoV-2 PCR testing done at dose 2 was not a pre-defined protocol objective. Due to the limited COVID-19 symptom data provided, it is unknown whether persons classified as asymptomatic experienced COVID-19 symptoms after dose 2 and were truly presymptomatic.

^m Very serious concern for indirectness was present. The intended outcome was asymptomatic infection assessed with serial PCR testing for SARS-CoV-2, to include follow-up after completion of the full vaccination schedule. Data are presented from an analysis of participants with SARS-CoV-2 positive PCR test results from nasopharyngeal swabs collected on the day of the second vaccine dose, among persons who were seronegative at baseline and did not report COVID-19 symptoms after dose 1. The available evidence are indirect because they represent 1) SARS-CoV-2 testing at a single point in time, 2) assessment after one dose, and 3) short follow-up period.

ⁿ Two small Phase I observational studies with no comparison group were not included in the GRADE analysis. In a total of 35 vaccinated persons, 0 serious adverse events were observed. Pooled estimates from the two RCTs were used for GRADE.

^o Serious concern of indirectness was noted. The body of evidence does not provide certainty that rare serious adverse events were captured due to the short duration of follow-up and the sample size.

^p Two small Phase I observational studies with no comparison group were not included in the GRADE analysis. In a total of 35 vaccinated persons, 2 grade 3 reactions were observed. Pooled estimates from the two RCTs were used for GRADE.

Appendix 1. Studies Included in the Review of Evidence23456910

Abbreviations: SD = standard deviation; RCT = randomized controlled trial; COVID-19 = coronavirus disease 2019.

Appendix 2. Databases and strategies used for systematic review