Grading of Recommendations, Assessment, Development, and Evaluation (GRADE): Janssen COVID-19 Vaccine

Overview

A Grading of Recommendations, Assessment, Development and Evaluation (GRADE) review of the evidence for benefits and harms for Janssen coronavirus disease 2019 (COVID-19) vaccine was presented to the Advisory Committee for Immunization Practices (ACIP) on February 28, 2021. GRADE evidence type indicates the certainty in estimates from the available body of evidence. Evidence certainty ranges from type 1 (high certainty) to type 4 (very low certainty).1

The policy question was, "Should vaccination with Janssen COVID-19 vaccine be recommended for persons 18 years of age and older during an Emergency Use Authorization?" The potential benefits pre-specified by the ACIP COVID-19 Vaccines Work Group were prevention of symptomatic laboratory-confirmed COVID-19 (critical), hospitalization due to COVID-19 (critical), all-cause death (important), SARS-CoV-2 seroconversion (important), and asymptomatic SARS-CoV-2 infection (important). The two pre-specified harms were serious adverse events (critical) and reactogenicity grade ≥3 (important).

A systematic review of evidence on the efficacy and safety of a one-dose regimen of Janssen COVID-19 vaccine among persons aged 18 years and older was conducted. The quality of evidence from one Phase I/II randomized controlled trial, one Phase II randomized controlled trial, and one Phase III randomized controlled trial were assessed using a modified GRADE approach.2345

The Phase III randomized controlled trial of the Janssen COVID-19 vaccine was conducted on three continents during a time of high COVID-19 incidence while viral variants were emerging. The trial exceeded the target numbers of cases to meet its primary endpoints by the time the required median 2-month follow-up was complete. The vaccine efficacy estimates for the Janssen COVID-19 vaccine were 66% for symptomatic laboratory-confirmed COVID-19, 74% for asymptomatic seroconversion, 93% for hospitalization due to COVID-19, and 75% for all-cause death. No deaths due to COVID-19 were identified among vaccine recipients, and 7 deaths due to COVID-19 were identified among placebo recipients. No serious safety concerns were identified in the randomized controlled trials, with balanced reports of serious adverse events between arms (0.4% each). Grade ≥3 local or systemic reactions were more common among vaccine than placebo recipients and were reported by <3% of vaccinated subjects. The certainty in the estimates for all critical benefits and harms was type 2 (moderate), with certainty impacted by concern for indirectness because of the short duration of follow-up in the available body of evidence.

Introduction

As of December 19, 2020, two vaccines had been recommended for prevention of COVID-19, caused by the SARS-CoV-2 virus which emerged in late 2019.67

On February 27, 2021, the U.S. Food and Drug Administration (FDA) issued an Emergency Use Authorization (EUA) for Janssen COVID-19 (Ad26.COV2.S) vaccine for prevention of symptomatic COVID-19 for persons aged ≥18 years.8 As part of the process employed by the ACIP, a systematic review and GRADE evaluation of the evidence for Janssen COVID-19 vaccine was conducted and presented to ACIP. There were no conflicts of interest reported by CDC and ACIP COVID-19 Vaccines Work Group members involved in the GRADE analysis.

The ACIP adopted a modified GRADE approach in 2010 as the framework for evaluating the scientific evidence that informs recommendations for vaccine use, including those for the Janssen COVID-19 vaccine. Evidence of benefits and harms were reviewed based on the GRADE approach.1

The primary policy question was, "Should vaccination with Janssen COVID-19 vaccine be recommended for persons 18 years of age and older during an Emergency Use Authorization?" (Table 1).

Methods

We conducted a systematic review of evidence on the efficacy and safety of a single-dose regimen of Janssen COVID-19 vaccine. We assessed outcomes and evaluated the quality of evidence using the GRADE approach.

We identified studies in Medline, Embase, and Cochrane Library, written in English, and limited to studies published from 2020 to February 8, 2021. Search terms included coronavirus, COVID-19, SARS-CoV-2, respiratory (symptom, disease, illness, condition), vaccine, immunization, trial, double blind, single blind, placebo, comparative study, phase I, phase II, phase III, immunogenicity, efficacy, effective, adverse, evidence, and variations on these terms (see Appendix 2 for details).

Articles were included if they provided data on vaccination with the Janssen COVID-19 vaccine and 1) involved human subjects; 2) reported primary data; 3) included adults (ages 18 and older) at risk for SARS-CoV-2 infection; 4) included data relevant to the efficacy and safety outcomes being measured; and 5) included data for the specific vaccine formulation, dosage, and schedule being recommended (Ad26.COV2.S, 5×10¹⁰ viral particles IM, single dose). In addition, efforts were made to obtain unpublished and other relevant data by hand-searching reference lists, and consulting with vaccine manufacturers and subject matter experts. Titles and abstracts were screened independently and in duplicate by two separate reviewers. Characteristics of the included studies are shown in Appendix 1.

Patient-important outcomes (including benefits and harms) for assessment were selected by the Work Group during Work Group calls and via online surveys where members were asked to rate and rank the importance of relevant outcomes. The GRADE assessment across the body of evidence for each outcome was presented in an evidence profile.

Relative risks (RR) were calculated from numerators and denominators available in the body of evidence. Vaccine efficacy estimates were defined as 100% x (1-RR). When multiple studies were available, pooled estimates were calculated using RevMan software.

Results

The results of the GRADE assessment were presented to ACIP on February 28, 2021.

After title and abstract screening of 3,999 records, 29 studies were identified as eligible for full-text review. Of these, 26 were excluded because they assessed a different vaccine. This left 3 studies for the evidence synthesis and GRADE evidence assessment.23 Characteristics of the included studies are shown in Appendix 1.

Outcomes of interest included individual benefits and harms. Benefits of interest deemed critical were prevention of symptomatic laboratory-confirmed COVID-19 and prevention of hospitalization due to COVID-19 (Table 2). Other important outcomes included prevention of all-cause death; SARS-CoV-2 seroconversion to a non-spike protein; and asymptomatic SARS-CoV-2 infection. The critical harm of interest was serious adverse events, including vaccine-associated enhanced disease; reactogenicity grade ≥3 was deemed an important harm. Asymptomatic infection was not included in the evidence profile because no data were available from serial PCR assessments.

Three studies were reviewed that provided data on outcomes specified for GRADE (Appendix 1). Data were reviewed from one published Phase I/II randomized control trial, with additional data provided by the sponsor.2 Data were reviewed from one unpublished Phase II randomized controlled trial and one Phase III randomized controlled trial using data provided by the sponsor and FDA.2345910

The Janssen COVID-19 vaccine reduced symptomatic laboratory-confirmed COVID-19 when compared to no COVID-19 vaccination (vaccine efficacy: 66%; 95% CI: 60%, 71%) (Table 3a). For hospitalization due to COVID-19 in cases with onset ≥14 days post vaccination, 31 events were documented, 2 in the placebo group and 29 in the vaccine group. Vaccine efficacy against hospitalization due to COVID-19 was 93% (95% CI: 71%, 98%) (Table 3b). All-cause deaths were less common, 5 in the vaccine group and 20 in the placebo group (VE: 75%; 95% CI: 33%, 91%) (Table 3c).

Preliminary data from serum collected at day 71 suggested a lower risk of seroconversion to a non-spike protein, i.e., marker of natural infection (vaccine efficacy 74%, 95% CI: 48%, 87%) (Table 3d).

For evaluation of potential harms, data were pooled from the Phase I/II, II, and Phase III randomized controlled trials. Proportions with serious adverse events were comparable between the vaccine group and the placebo group when pooled across the three studies (pooled RR: 0.85; 95% CI: 0.63, 1.13); there were no cases of vaccine-associated enhanced disease or vaccine-related deaths (Table 4 (pooled), Table 3e (individual trial data)). Grade ≥3, or severe, local or systemic reactions 7 days following vaccination, were reported by 2.5% of vaccine recipients and 0.7% of placebo recipients (Table 4 (pooled), Table 3f (individual trial data)). No grade 4 reactions were reported.

GRADE Summary

The initial GRADE evidence level was type 1 (high) for each outcome because the body of evidence was from randomized controlled trials (Table 4). In terms of critical benefits, the available data indicated that the vaccine was effective for preventing symptomatic COVID-19 and for preventing hospitalization due to COVID-19; these were each downgraded one point for serious concern of indirectness related to the median two months follow-up (type 2, moderate). The certainty in the effect estimate for the important outcome of all-cause death was downgraded one point for serious concern of indirectness related to the median two months follow-up (type 2, moderate). The certainty in the estimate for seroconversion was downgraded two points due to very serious concern of indirectness because the efficacy against seroconversion based on day 71 serology may not be a direct measure of efficacy over a relevant period of time for an emergency use authorization and the subset of 6.8% with serology data likely differed substantially from the total population of interest (type 3, low). The certainty in the estimate of the effect for serious adverse events was downgraded one point due to serious concern of indirectness related to the median two months follow-up (type 2, moderate). No serious concerns impacted the certainty in the estimate of reactogenicity (type 1, high) (Table 4).

Table 1: Policy Question and PICO

Abbreviations: IM = intramuscular.

^a Assessed through serial PCR testing.

Table 2: Outcomes and Rankings

^a Data from serial PCR assessments to inform an evaluation of asymptomatic SARS-CoV-2 infection were not available in studies identified in the review of evidence.

Table 3a: Summary of Studies Reporting Symptomatic Laboratory-confirmed COVID-19^a

References in this table: 3

Abbreviations: RT-PCR = real-time polymerase chain reaction; CI = confidence interval; RR = relative risk.

^a 21895 and 21888 persons were randomized to vaccine and placebo

^b Primary outcome, defined as SARS-CoV-2 RT-PCR-positive symptomatic illness, in seronegative adults, ≥14 days post vaccination.

^c Symptomatic illness defined as at least one nasopharyngeal swab, nasal swab, or saliva sample (or respiratory sample, if hospitalized) positive for SARS-CoV-2 by RT-PCR; and at least two symptoms: fever (≥ 38ºC), chills, myalgia, headache, sore throat, new olfactory and taste disorder; or at least one symptom: cough, shortness of breath or difficulty breathing, or clinical or radiographical evidence of pneumonia.

Based on data cutoff January 22, 2021; participants had a median of two months follow-up.

Table 3b: Summary of Studies Reporting Hospitalization due to COVID-19 ^a

References in this table: 3

Abbreviations: CI = confidence interval; RR = relative risk

^a Data on hospitalizations was obtained from Medical Resource Utilization form or Serious Adverse Events form.

^b Participants aged ≥18 years, without evidence of prior infection, ≥14 days after vaccination.

Based on data cutoff January 22, 2021.

Table 3c: Summary of Studies Reporting All-cause Death ^a

References in this table: 3

Abbreviations: RR = relative risk; CI = confidence interval.

^a Death from any cause, including COVID-related or serious adverse event.

Based on data collected through February 5, 2021.

Table 3d: Summary of Studies Reporting Asymptomatic SARS-CoV-2 Infection^a

References in this table: 3

Abbreviations: RR = relative risk; CI = confidence interval.

^a Asymptomatic SARS-CoV-2 infection is defined as (1) positive serology (non-S protein), and (2) no prior SARS-CoV-2 positive PCR or COVID-19 symptoms during the study. Seroconversion to a non-spike protein can distinguish between natural infection and vaccine-induced immunity.

Table 3e: Summary of Studies Reporting Serious Adverse Events^{a, b}

References in this table: 23

Abbreviations: RR = relative risk; CI = confidence interval; RCT = randomized controlled trial.

^a Death, life-threatening event, hospitalization, incapacity to perform normal life functions, medically important event, or congenital anomaly/birth defect.

^b Excludes serious adverse events due to COVID-19.

^c 9 participants (7 in the vaccine and 2 in the placebo group) were deemed by blinded investigators to have serious adverse events related or possibly related to vaccination. Among the 7 vaccine participants, these included: pericarditis, facial paralysis, injection site pain, Guillain-Barre Syndrome, systemic reactogenicity, and hypersensitivity. Through further investigation by the FDA, only 3 were classified as likely related to vaccination: injection site pain, hypersensitivity, and systemic reactogenicity.

Table 3f: Summary of Studies Reporting Reactogenicity^a

References in this table: 23

Abbreviations: RR = relative risk; CI = confidence interval.

^a Reactogenicity outcome includes local and systemic events, grade ≥3. Grade 3: prevents daily routine activity or requires use of a narcotic pain reliever. Grade 4: requires emergency room visit or hospitalization. No participants reported grade 4 reactions.

Based on interim analysis, data cutoff January 22, 2021.

Table 4. Grade Summary of Findings Table

Abbreviations: CI = confidence interval; RR = relative risk; COVID-19 = coronavirus disease 2019; RCT = randomized controlled trial.

a. Risk of bias related to blinding of participants and personnel may have been present. Although participants and study staff were blinded to intervention assignments, they may have inferred receipt of vaccine or placebo based on reactogenicity. This was deemed unlikely to overestimate efficacy or underestimate risk of serious adverse events, therefore the risk of bias was rated as not serious.

b. Serious concern for indirectness was noted due to the short duration of observation in the available body of evidence. Vaccine efficacy or adverse events observed at a median 2-month follow-up may differ from those observed with ongoing follow-up.

c. The effects noted are from an analysis of the per protocol population with outcomes assessed at least 14 days post vaccination, who had no evidence of prior SARS-CoV-2 infection, and counting cases who met the case definition with symptoms for moderate to severe COVID-19 and were PCR positive but not necessarily molecularly confirmed at the central laboratory. In an interim analysis using the full analysis set (persons with or without evidence of prior SARS-CoV-2 infection), there were 267 cases among 21,895 persons in the vaccine arm and 621 cases among 21,888 persons in the placebo arm (RR = 0.43 (0.37 to 0.50)).

d. The RCT excluded persons with prior COVID-19 diagnosis, pregnant women or women planning to become pregnant within 3 months of vaccination, and persons who were immunocompromised due to conditions and/or treatments (participants with stable/well-controlled HIV infection were not excluded). The population included in the RCT may not represent all persons aged >=18 years.

e. Absolute risk was calculated using the observed outcomes in the placebo arm during the available clinical trial follow-up. Absolute risk estimates should be interpreted in this context.

f. The effects noted are from an analysis of the per protocol population with outcomes assessed at least 14 days post vaccination among persons who had no evidence of prior SARS-CoV-2 infection.

g. Includes 15 hospitalized cases in the placebo arm identified from the Serious Adverse Events form rather than from the Medical Resource Utilization form.

h. Very serious concern for indirectness was noted. Efficacy against seroconversion based on day 71 serology may not be a direct measure of efficacy over a relevant period of time for an emergency use authorization. Additionally, serology data were only available for a subset of 7% of the per protocol population, likely not representing all ages, comorbidities, geographies, and exposures to circulating variants, raising additional concern for indirectness.

i. Data were pooled from one Phase III trial, one Phase I/II trial, and one Phase II trial.

j. Differences in demographic composition were noted between the full analysis set (FAS) and the safety subset used for evaluation of reactogenicity. Notably, compared to the FAS, the safety subset included a higher proportion of White race (83.4% vs. 58.7%), a higher proportion from Brazil (38.5% vs. 16.6%), and a lower proportion who were seropositive for SARS-CoV-2 at baseline (4.5% vs. 9.6%).

Appendix 1. Studies Included in the Review of Evidence

References in this table: 2345910

Abbreviations: SD = standard deviation; RCT = randomized controlled trial; COVID-19 = coronavirus disease 2019.

Appendix 2. Databases and strategies used for systematic review

a. Most recent search conducted February 8, 2021.