Interpretation of the results of a review of NRS must include consideration of the likely direction and magnitude of bias. Biases that affect randomized trials also affect NRS but typically to a greater extent. For example, attrition in NRS is often worse (and poorly reported), intervention and outcome assessment are rarely conducted according to standardized protocols, and outcomes are rarely blind. Too often these limitations of NRS are seen as part of doing a NRS, and their implications for risk of bias are not properly considered. For example, some users of evidence may consider NRS that investigate long-term outcomes to have ‘better quality’ than randomized trials of short-term outcomes, simply on the basis of their relevance without appraising their risk of bias (see Section 13.2.1.4).
Assessing the magnitude of confounding in NRS is especially problematic. Review authors must not only have adequate methods for assessment but also collect and report adequate detail about the confounding factors considered by researchers and the methods used to control for confounding. The information may not be available from the reports of the primary studies, preventing the review authors from investigating differences in the methods of eligible studies and other sources of heterogeneity that were considered likely to be important when the protocol was written.
Authors must remember the following points about confounding:
The direction of the bias introduced by confounding is unpredictable;
Methods used by researchers to control for confounding are like to vary between studies;
The extent of residual confounding in any particular study is unknown, and is likely to vary between studies;
Residual confounding (and other biases) means that confidence intervals underestimate the true uncertainty around an effect estimate.
It is important to identify the likely confounding factors that have not been adjusted for, as well as those that have been adjusted for.
The challenges described above affect all systematic reviews of NRS. However, challenges may be less extreme in some healthcare areas (e.g. confounding may be less of a problem in observational studies of long-term or adverse effects, or some public health primary prevention interventions).
One clue to the presence of bias is notable between-study heterogeneity. Although heterogeneity can arise through differences in participants, interventions and outcome assessments, the possibility that bias is the cause of heterogeneity in reviews of NRS must be considered seriously. However, lack of heterogeneity does not indicate lack of bias, since it is possible that a consistent bias applies in all studies.
Can the magnitude and direction of bias be predicted? This is a subject of ongoing research which is attempting to gather empirical evidence on factors (such as study design and intervention type) that determine the size and direction of these biases. The ability to predict both the likely magnitude of bias and the likely direction of bias would greatly improve the usefulness of evidence from systematic reviews of NRS. There is currently some evidence that in some limited circumstances the direction, at least, can be predicted (Henry 2001)