Decisions about which studies to include in a review are among the most influential decisions that are made in the review process. However, they involve judgement. To help ensure that these judgements are reproducible, it is desirable for more than one author to repeat parts of the process. In practice, the exact approach may vary from review to review, depending in part on the experience and expertise of the review authors.
Authors must first decide if more than one of them will assess the titles and abstracts of records retrieved from the search (step 2 in Section 7.2.3). Using at least two authors may reduce the possibility that relevant reports will be discarded (Edwards 2002). It is most important that the final selection of studies into the review is undertaken by more than one author (step 5 in Section 7.2.3).
Experts in a particular area frequently have pre-formed opinions that can bias their assessments of both the relevance and validity of articles (Cooper 1989, Oxman 1993). Thus, while it is important that at least one author is knowledgeable in the area under review, it may be an advantage to have a second author who is not a content expert. Some authors may decide that assessments of relevance should be made by people who are blind or masked to information about the article, such as the journal that published it, the authors, the institution, and the magnitude and direction of the results. They could attempt to do this by editing copies of the articles. However, this takes much time, and may not be warranted given the resources required and the uncertain benefit in terms of protecting against bias (Berlin 1997).
Disagreements about whether a study should be included can generally be resolved by discussion. Often the cause of disagreement is a simple oversight on the part of one of the review authors. When the disagreement is due to a difference in interpretation, this may require arbitration by another person. Occasionally, it will not be possible to resolve disagreements about whether to include a study without additional information. In these cases, authors may choose to categorize the study in their review as one that is awaiting assessment until the additional information is obtained from the study authors.
In summary, the methods section of both the protocol and the review should detail:
whether more than one author examines each title and abstract to exclude obviously irrelevant reports;
whether those who examine each full-text report to determine eligibility will do so independently (this should be done by at least two people);
whether the decisions on the above are made by content area experts, methodologists, or both;
whether the people assessing the relevance of studies know the names of the authors, institutions, journal of publication and results when they apply the eligibility criteria; and
how disagreements are handled.
A single failed eligibility criterion is sufficient for a study to be excluded from a review. In practice, therefore, eligibility criteria for each study should be assessed in order of importance, so that the first ‘no’ response can be used as the primary reason for exclusion of the study, and the remaining criteria need not be assessed.
For most reviews it will be worthwhile to pilot test the eligibility criteria on a sample of reports (say ten to twelve papers, including ones that are thought to be definitely eligible, definitely not eligible and doubtful). The pilot test can be used to refine and clarify the eligibility criteria, train the people who will be applying them and ensure that the criteria can be applied consistently by more than one person.