A properly conducted systematic review defines the question to be addressed in advance of the identification of potentially eligible trials. Systematic reviews are by nature, however, retrospective because the trials included are usually identified after the trials have been completed and the results reported (Pogue 1998, Zanchetti 1998). Knowledge of the results of individual randomized trials may introduce bias into a retrospective systematic review if the selection of the key components of the review question is based on reports of one or more positive trials. This might include influencing:
the criteria for study selection (i.e. the types of trial considered eligible);
the selection of the target population;
the nature of the intervention;
the choice of comparator; and
the outcomes to be assessed and their measures.
Take, for example, a systematic review in which the results of one study are in the opposite direction to those of the other studies in the review. The authors of the review discuss possible explanations for this apparent heterogeneity and decide that there is a clinical explanation. On this basis, the authors subsequently decide to exclude the study. This may be a reasonable decision; however, it is one made after the effect of the study’s results on the overall summary estimate is known, and hence is intrinsically problematic.
As described in detail in Chapter 10 (Section 10.2), awareness of the results of a trial may also influence the decision to publish those results. Even within a published trial, results may be selectively reported, thereby introducing a more subtle form of publication bias into the review (Chan 2004).
A prospective meta-analysis (PMA) is a meta-analysis of studies (usually randomized trials) that were identified, evaluated and determined to be eligible for the meta-analysis before the results of any of those studies became known. They have features in common with both cumulative meta-analyses and those involving individual patient data (Egger 1997). PMA can help to overcome some of the recognized problems of retrospective meta-analyses (see also Chapter 18, Section 18.5) by:
enabling hypotheses to be specified a priori ignorant of the results of individual trials;
enabling prospective application of study selection criteria; and
enabling a priori statements of intended analyses, including subgroup analyses, to be made before the results of individual trials are known. This avoids potential difficulties in interpretation related to the data-dependent emphasis on particular subgroups.
Systematic reviews also depend on the ability of the review authors to obtain data on all randomized patients for the relevant outcomes, which can be difficult if full information is not reported in the trial publications. As most PMAs will collect and analyse individual patient data (IPD) they will be able to overcome this problem, with the additional advantage of being able to conduct time-to-event analyses if appropriate. Planned subgroup analyses based on patient-level factors can give misleading results if relying only on aggregate-level data, highlighting another advantage of IPD. PMA also provides a unique opportunity for trial design, data collection and other clinical trial processes to be standardized across trials. For example, the investigators may agree to use the same instrument to measure a particular outcome, and to measure the outcome at the same time-points in each trial. In a Cochrane review of interventions for preventing obesity in children, for example, the heterogeneity and unreliability of the some of the outcome measures made it difficult to pool data across trials (Summerbell 2005). A prospective meta-analysis of this question has proposed a set of commonly shared standards, so that some of the issues raised by lack of standardization can be addressed (Steinbeck 2006).