Chapter 5: Choosing and Ranking Outcomes

Summary

In order to develop PICO questions for a systematic review, outcomes that are considered 'critical' or 'important' for decision-making to inform the recommendation need to be identified1. Outcomes should be selected based on relevance to the target population (i.e., who the guideline will benefit) and considerations of stakeholders. The outcome selection process should be conducted ideally during the protocol development stage and before the evidence is reviewed to avoid listing only outcomes measured in the existing literature; outcomes should focus on what might be critical and important in the decision-making process. Outcomes that are not addressed in the literature should not be disregarded because they may still influence the recommendation and help identify knowledge gaps. Searching the literature or surveying stakeholders can help to identify patient-important outcomes; however, it's not necessary, as these may also be informed by the guideline development panel or work group. The important aspect of identifying outcomes is that they are representative of those that the benefactors of the recommendation would be likely to weigh when deciding about whether or not to choose a specific course of action.

When evidence about a particular population-important outcome is very limited, surrogate outcomes may be considered to inform the health outcome. A surrogate or intermediate outcome refers to an indicator which serves as a measurement of a clinically meaningful outcome (e.g., anti-diphtheria toxoid antibody level as a surrogate outcome to the induction of immunity in individuals immunized against diphtheria). However, the population-important outcome for which the surrogate outcome is substituting for should be specified and considered when grading the certainty of the evidence. If a surrogate outcome is used, the importance of the corresponding health outcome (e.g., prevention of orthopoxviral disease) rather than that of the surrogate outcome (e.g., neutralizing antibody response or IgG levels) should be scored (Figure 2). The surrogate should not be listed as the health outcome of interest. Figure 2 shows a list of patient-important outcomes with surrogate outcomes mentioned separately to demonstrate that while they can be used when evidence is limited, they are not the outcome of interest. Use of a surrogate outcome requires assessment of the level of indirectness in informing the health outcome. When multiple surrogate outcomes are available, the least indirect to the health outcome should be assessed first to reduce the extent of potential indirectness2.

After developing a list of relevant outcomes, the importance of these outcomes should be rated and ranked3. ACIP workgroup members should make an initial list of all possible relevant outcomes, including both desirable and undesirable effects. Each member should be asked to rate (score) the importance of each outcome on a 1 to 9 scale using a modified Delphi process, where 7–9 indicates that the outcome is critical for a decision, 4–6 indicates that it is important but not critical, and 1–3 indicates that it is of limited importance. Survey software (e.g., Google Forms, SurveyMonkey or Polls in Zoom) may be used to facilitate this process. The mean score for each outcome can be used to determine its relative importance, though it is helpful to provide the range of results as well, as this can give insight into any possible misunderstandings or divisions that warrant further discussion before finalizing the list of outcomes. Figure 2 provides a visual representation of the scale4. Then the workgroup members will rank the highest rated outcomes so that the top 5–7 outcomes are included in the evidence review and evidence profile. It's worth noting that among the 5–7 outcomes included in the grading process, at least one must be an undesirable effect. Table 2a and 2b provides examples of outcome lists with initial rankings.

Figure 2. Theoretical example of Listing and Ranking Outcomes using the modified Delphi Process based on previously published guidelines for use of smallpox vaccine in laboratory and health-care personnel at risk for occupational exposure to orthopoxviruses [adapted]

References in this figure: 345

View Larger

Table 2a. Example of Outcomes and Rankings

References in this table:6

*Three options: 1. Critical for decision making; 2. Important but not critical for decision making; 3. Not important for decision making.

Table 2b. Examples of Outcomes and Ratings from “Use of JYNNEOS (orthopoxvirus) vaccine primary series for research, clinical laboratory, response team, and healthcare personnel (Policy Questions 1 and 2)"

References in this table:7

*Three options: 1. Critical for decision making; 2. Important but not critical for decision making; 3. Not important for decision making.
**Prevention of disease was informed by the surrogate outcome of geometric mean titer.
***Serious adverse events were defined according to the standard FDA definition. In addition, data was collected about any smallpox vaccine-specific adverse event: postvaccinal encephalitis, eczema vaccinatum, progressive vaccinia, and generalized vaccinia.

There are three points in time that the outcome rating can be updated as either critical, important or not important for decision-making. First, outcomes are listed and ranked before the systematic review search is conducted during the brainstorming phase. Second, the rating of the outcomes can be updated after the evidence has been retrieved if the literature provides rationale for why an outcome ranking needs to be altered. Lastly, when making the recommendation, the outcome ratings can be changed again if needed.

After the evidence review is conducted, outcome ranking can be reassessed because in some cases the importance of an outcome is better known after considering the literature. For instance:

• An outcome pertaining to a benefit may have been judged initially to be critical for making a recommendation, but it may no longer be considered to be critical if other benefits are evident, or;

• A suspected adverse event may be initially considered to be critical, but if the evidence review shows that the adverse event is not causally associated with the intervention, it may be considered important but not critical.

When creating evidence profiles, important and critical outcomes should be included even if no literature is available about them. The guideline recommendation will primarily be influenced by critical outcomes.

There may be a situation in which evidence, including surrogate or indirect evidence, is not identified to inform one or more of the critical or important outcomes that are set to be presented in the evidence profile. One solution is to recognize there is no evidence to present and enter a "-" (dash) in each box within the evidence profile (including the certainty of evidence). In GRADEpro GDT, when editing the specific outcome there is an option "Not reported" which will autofill this for you. This transparently presents the results from the search and allows readers to recognize that future research may be needed to inform this outcome. If the work group feels that they can move forward with a recommendation in the absence of this specific outcome, the lack of evidence for this specific outcome will not influence the overall certainty of the evidence.