When risks of bias vary across studies in a meta-analysis, three broad strategies are available for choosing which result to present as the main finding for a particular outcome (for instance, in deciding which result to present in the Abstract). The intended strategy should be described in the protocol for the review.
The first approach involves defining a threshold, based on key bias domains (see Section 8.7) such that only studies meeting specific criteria are included in the primary analysis. The threshold may be determined using the original review eligibility criteria, or using reasoned argument (which may draw on empirical evidence of bias from meta-epidemiological studies). In rare cases, within-meta-analysis comparisons of studies at high and low risk of bias may produce evidence of differences between intervention effect estimates and justify restricting analyses to studies at low risk of bias (see Section 8.8.2.3). If the primary analysis includes studies at unclear risk of bias, review authors should justify this choice. Ideally the threshold, or the method for determining it, should be specified in the review protocol. Authors should keep in mind that all thresholds are arbitrary, and that studies may in theory lie anywhere on the spectrum from ‘free of bias’ to ‘undoubtedly biased’. The higher the threshold, the more similar the studies will be in their risks of bias, but they may end up being few in number. Review authors who restrict their primary analysis in this way are encouraged to perform sensitivity analyses showing how conclusions might be affected if studies at high risk of bias were included.
Stratifying on the summary risk of bias may produce at least three estimates of the intervention effect: from studies at high and low risk of bias and from all studies. Two or more such estimates might be presented with equal prominence, for example, one including all studies and one including only those at low risk of bias. This avoids the need to make a difficult decision, but may be confusing for readers. In particular, people who need to make a decision usually require a single estimate of effect. Further, ‘Summary of findings’ tables will usually only present a single result for each outcome. On the other hand, a stratified forest plot presents all information transparently.
The choice between strategies 1 and 2 should be based on the context of the particular review and the balance between the potential for bias and the loss of precision when studies at high or unclear risk of bias are excluded. As explained in Section 8.8.2.3, lack of a statistically significant difference between studies at high and low risk of bias should not be interpreted as implying absence of bias, because meta-regression analyses typically have low power.
The simplest approach to incorporating bias assessments in results is to present an estimated intervention effect based on all available studies, together with a description of the risk of bias in individual domains, or a description of the summary risk of bias, across studies. This is the only feasible option when all studies are at high risk, all are at unclear risk or all are at low risk of bias. However, when studies have different risks of bias, we discourage such an approach for two reasons. First, detailed descriptions of risk of bias in the results section, together with a cautious interpretation in the discussion section, will often be lost in the conclusions, abstract and summary of findings, so that the final interpretation ignores the risk of bias and decisions continue to be based, at least in part, on flawed evidence. Second, such an analysis fails to down-weight studies at high risk of bias and hence will lead to an overall intervention that is too precise as well as being potentially biased.
When the primary analysis is based on all studies, summary assessments of risk of bias must be incorporated into explicit, measures of the quality of evidence for each important outcome, for example using the GRADE system (Guyatt 2008). This can help to ensure that judgements about the risk of bias, as well as other factors affecting the quality of evidence, such as imprecision, heterogeneity and publication bias, are appropriately taken into consideration in interpreting the results of the review (See Chapter 11, Section 11.5 and Chapter 12, Section 12.2).