When more than one author extracts data from the same reports, there is potential for disagreement. An explicit procedure or decision rule should be identified in the protocol for identifying and resolving disagreements. Most often, the source of the disagreement is an error by one of the extractors and is easily resolved. Thus, discussion among the authors is a sensible first step. More rarely, a disagreement may require arbitration by another person. Any disagreements that cannot be resolved should be addressed by contacting the study authors; if this is unsuccessful, the disagreement should be reported in the review.
The presence and resolution of disagreements should be carefully recorded. Maintaining a copy of the data ‘as extracted’ (in addition to the consensus data) allows assessment of reliability of coding. Examples of ways in which this can be achieved include:
Use one author’s (paper) data collection form and record changes after consensus in a different ink colour.
Use a separate (paper) form for consensus data.
Enter consensus data onto an electronic form.
Agreement of coded items can be quantified, for example using kappa statistics (Orwin 1994), although this is not routinely done in Cochrane reviews. A simple calculation for agreement between two authors is described in Section 7.2.6. If agreement is assessed, this should be done only for the most important data (e.g. key risk of bias assessments, or availability of key outcomes).
Informal consideration of the reliability of data extraction should be borne in mind throughout the review process, however. For example, if after reaching consensus on the first few studies, the authors note a frequent disagreement for specific data, then coding instructions may need modification. Furthermore, an author’s coding strategy may change over time, as the coding rules are forgotten, indicating a need for re-training and, possibly, some re-coding.