The main concerns over risk of bias in cross-over trials are: (i) whether the cross-over design is suitable; (ii) whether there is a carry-over effect; (iii) whether only first period data are available; (iv) incorrect analysis; and (v) comparability of results with those from parallel-group trials.
(i) The cross-over design is suitable to study a condition that is (reasonably) stable (e.g. asthma), and where long-term follow-up is not required. The first issue to consider therefore is whether the cross-over design is suitable for the condition being studied.
(ii) Of particular concern is the possibility of a ‘carry over’ of treatment effect from one period to the next. A carry-over effect means that the observed difference between the treatments depends upon the order in which they were received; hence the estimated overall treatment effect will be affected (usually underestimated, leading to a bias towards the null).
The use of the cross-over design should thus be restricted to situations in which there is unlikely to be carry-over of treatment effect across periods. Support for this notion may not be available, however, before the trial is done. Review authors should seek information in trial reports about the evaluation of the carry-over effect. However, in an unpublished review of 116 published cross-over trials from 2000 (Mills 2005), 30% of the studies discussed carry-over but only 12% reported the analysis.
(iii) In the presence of carry-over, a common strategy is to base the analysis on only the first period. Although the first period of a cross-over trial is in effect a parallel group comparison, use of data from only the first period will be biased if, as is likely, the decision to do so is based on a test of carry-over. Such a ‘two stage analysis’ has been discredited (Freeman 1989) but is still used. Also, use of the first period only removes the main strength of the cross-over design, the ability to compare treatments within individuals.
Cross-over trials for which only first-period data are available should be considered to be at risk of bias, especially when the investigators explicitly used the two-stage strategy.
(iv) The analysis of a cross-over trial should take advantage of the within-person design, and use some form of paired analysis (Elbourne 2002). Although trial authors may have analysed paired data, poor presentation may make it impossible for review authors to extract paired data. Unpaired data may be available and will generally be unrelated to the estimated treatment effect or statistical significance. So it is not a source of bias, but rather will usually lead to a trial getting (much) less than its due weight in a meta-analysis.
In the review above (Mills 2005), only 38% of 116 cross-over trials performed an analysis of paired data.
(v) In the absence of carry-over, cross-over trials should estimate the same treatment effect as parallel group trials. Although one study reported a difference in the treatment effect found in cross-over trials compared with parallel group trials (Khan 1996), they had looked at treatments for infertility, an area notorious for the inappropriateness of the cross-over design, and a careful re-analysis did not support the original findings (te Velde 1998).
Other issues to consider for risk of bias in cross-over trials include the following.
Participants may drop out after the first treatment, and not receive the second treatment. Such participants are usually dropped from the analysis.
There may be a systematic difference between the two periods of the trial. A period effect is not too serious, as it applies equally to both treatments, although it may suggest that the condition being studied is not stable.
It may not be clear how many treatments or periods were used. Lee could not identify the design for 12/64 published cross-over trials (Lee 2005b).
It should not be assumed that the order of treatments was randomized in a cross-over trial. Occasionally a study may be encountered in which it is clear that all participants had the treatments in the same order. Such a trial does not provide a valid comparison of the treatments, since there may be a trend in outcomes over time in addition to the change in treatments.
Reporting of drop-outs may be poor, especially for those participants who completed one treatment period. The number of participants who dropped out was specified in only nine of the 64 trials in Lee’s review (Lee 2005b).
Some suggested questions for assessing risk of bias in cross-over trials are as follows:
Was use of a cross-over design appropriate?
Is it clear that the order of receiving treatments was randomized?
Can it be assumed that the trial was not biased from carry-over effects?
Are unbiased data available?