Although the general advice is to assess risk of bias in clinical trials as described in Chapter 8, authors must also consider other specific factors that may have a larger influence on the adverse effects data. Areas of special concern include methods for monitoring and detecting adverse effects, conflicting interests (Jüni 2004), selective outcome reporting (Chan 2004) and blinding (Schulz 2002).
The primary outcome measure of an intervention may have been studied in a placebo controlled, well-masked, adequately concealed randomized trial. In contrast, the adverse effects data may be collected retrospectively, for example via an end-of-study questionnaire sent out only to those who are known to have received the active intervention. Although a low risk of bias may be assigned to the primary outcomes, the way in which harmful effects of the interventions are monitored may not permit a similar rating. The recommended risk of bias tool, implemented in RevMan, allows for different assessments of blinding and of incomplete outcome data for each outcome, or for a class of outcomes as defined by the review author.
The methods used in monitoring or detecting adverse effects are known to have a major influence on adverse effect frequencies: studies in which adverse effects are carefully sought will report a higher frequency than studies in which they are sought less carefully. For example, in a group of hypertensive patients, passive monitoring based on spontaneous reports yielded rates of 16%, while active surveillance using specific questioning found a rate of 62% (Olsen 1999). As different methods of monitoring adverse effects will yield different results, it may be difficult to compare studies, and pointless to do a formal meta-analysis (Edwards 1999). Duration and frequency of monitoring should also be noted.
Studies with limited follow-up or infrequent monitoring may not reliably detect adverse effects; the absence of information must not be interpreted as indicating the intervention is safe. In contrast, studies with rigorous follow-up and active surveillance for pre-defined adverse effects may be able to generate evidence that the intervention genuinely has few adverse effects.
Finally, the age of an intervention and the evolution of its use are likely to be related to the types of adverse events detected and their number. This is obvious for long-term effects such as carcinogenicity, but also because some interventions, for example in surgery, change more or less subtly over time.
Examples of potentially useful questions to consider in assessing the quality of evidence on adverse effects are:
On conduct:
Are definitions of reported adverse effects given?
Were the methods used for monitoring adverse effects reported? Use of prospective or routine monitoring; spontaneous reporting; patient checklist, questionnaire or diary; systematic survey of patients?
On reporting:
Were any patients excluded from the adverse effects analysis?
Does the report provide numerical data by intervention group?
Which categories of adverse effects were reported by the investigators?