“A leap of faith is always required when applying any study findings to the population at large” or to a specific person. “In making that jump, one must always strike a balance between making justifiable broad generalizations and being too conservative in one’s conclusions” (Friedman 1985).
To address adequately the extent to which a review is relevant for the purpose to which it is being put (‘directness’), there are certain things the review author must do, and certain things the user of the review must do. We discuss here what the review author can do to help the user. Cochrane review authors must be extremely clear on the population, intervention, and outcomes that they are intending to address. Chapter 11 (Section 11.5.2) emphasizes a crucial step that has not traditionally been part of Cochrane reviews: the specification of all patient-important outcomes relevant to the intervention strategies under comparison.
With respect to participant and intervention factors, review authors need to make a priori hypotheses about possible effect modifiers, and then examine those hypotheses. If they find apparent subgroup effects, they must ultimately decide whether or not these effects are credible (Oxman 2002). Differences between subgroups, particularly those that correspond to differences between studies, need to be interpreted cautiously. Some chance variation between subgroups is inevitable, so unless there is strong evidence of an interaction authors should not assume that the subgroup effect exists. If, despite due caution, review authors judge subgroup effects as credible, they should conduct separate meta-analyses for the relevant subgroups, and produce separate ‘Summary of findings’ tables for those subgroups.
The user of the review will be challenged with ‘individualization’ of the findings. For example, even if relative effects are similar across subgroups, absolute effects will differ according to baseline risk. Review authors can help provide this information by identifying identifiable groups of people with varying risks in the ‘Summary of findings’ tables, as discussed in Chapter 11 (Section 11.5.5). Users can then identify the patients before them as belonging to a particular risk group, and assess their likely magnitude of benefit or harm accordingly.
Another decision users must make is whether the patients before them are so different from those included in the studies that they cannot use the results of the systematic review and meta-analysis at all. Review authors can point out that, rather than rigidly applying the inclusion and exclusion criteria of studies, it is better to ask whether there are compelling reasons why the evidence should not be applied to a particular patient (Guyatt 1994). Authors can sometimes help clinical decision makers by identifying important variation where divergence might limit the applicability of results (Schünemann 2006a), including: biologic and cultural variation, and variation in adherence to an intervention.
In addressing these issues, authors cannot be aware of, or address, the myriad of differences in circumstances around the world. They can, however, address differences of known importance to many people and, importantly, they should avoid assuming that other people's circumstances are the same as their own in discussing the results and drawing conclusions.