Some general guidelines for determining suitable topics for assessment as ‘other sources of bias’ are provided below. In particular, suitable topics should constitute potential sources of bias and not sources of imprecision, sources of diversity (heterogeneity) or measures of research quality that are unrelated to bias. The topics covered in this domain of the tool include primarily the examples provided in Section 8.15.1. Beyond these specific issues, however, review authors should be alert for study-specific issues that may raise concerns about the possibility of bias, and should formulate judgements about them under this domain of the tool. The following considerations may help review authors assess whether a study is free of risk of bias from other sources using the Collaboration’s tool (Section 8.5).
Wherever possible, a review protocol should pre-specify any questions to be addressed, which would lead to separate entries in the ‘Risk of bias’ table. For example, if cross-over trials are the usual study design for the question being addressed by the review, then specific questions related to bias in cross-over trials should be formulated in advance.
Issues covered by the risk of bias tool must be a potential source of bias, and not just a cause of imprecision (see Section 8.2), and this applies to aspects that are assessed under this ‘other sources of bias’ domain. A potential source of bias must be able to change the magnitude of the effect estimate, whereas sources of imprecision affect only the uncertainty in the estimate (i.e. its confidence interval). Potential factors affecting precision of an estimate include technological variability (e.g. measurement error) and observer variability.
Because the tool addresses only internal biases, any issue covered by this domain should be a potential source of internal bias, and not a source of diversity. Possible causes of diversity include differences in dose of drug, length of follow-up, and characteristics of participants (e.g. age, stage of disease). Studies may select doses that favour the experimental drug over the control drug. For example, old drugs are often overdosed (Safer 2002) or may be given under clearly suboptimal circumstances that do not reflect clinical practice (Johansen 2000, Jørgensen 2007). Alternatively, participants may be selectively chosen for inclusion in a study on the basis of previously demonstrated ‘response’ to the experimental intervention. It is important that such biased choices are addressed in Cochrane reviews. Although they may not be covered by the ‘Risk of bias’ tool described in the current chapter, they may sometimes be addressed in the analysis (e.g. by subgroup analysis and meta-regression) and should be considered in the grading and interpretation of evidence in a ‘Summary of findings’ table (see Chapters 11 and 12).
Many judgements can be made about the design and conduct of a clinical trial, but not all of them may be associated with bias. Measures of ‘quality’ alone are often strongly associated with aspects that could introduce bias. However, review authors should focus on the mechanisms that lead to bias rather than descriptors of studies that reflect only ‘quality’. Some examples of ‘quality’ indicators that should not be assessed within this domain include criteria related to applicability, ‘generalizability’ or ‘external validity' (including those noted above), criteria related to precision (e.g. sample size or use of a sample size (or power) calculation), reporting standards, and ethical criteria (e.g. whether the study had ethical approval or participants gave informed consent). Such factors may be important, and would be presented in the table of ‘Characteristics of included studies’ or in Additional tables (see Chapter 11)
Finally, to avoid double-counting, potential sources of bias should not be included as ‘bias from other sources’ if they are more appropriately covered by earlier domains in the tool. For example, in Alzheimer’s disease, patients deteriorate significantly over time during the trial. Generally, the effects of treatments are small and treatments have appreciable toxicity. Dealing satisfactorily with participant losses is very difficult. Those on treatment are likely to drop out earlier due to adverse effects or death, and hence the measurements on these people, tending to be earlier in the study, will favour the intervention. It is often difficult to get continued monitoring of these participants in order to carry out an analysis of all randomized participants. This issue, although it might at first seem to be a topic-specific cause of bias, would be more appropriately covered under Incomplete Outcome Data.