Assessment of study quality (critical appraisal) is a particularly contested issue in relation to qualitative evidence synthesis. At present, opinion on the value of formal quality assessment is divided and there is insufficient evidence to inform a judgement on the rigour or added value of various approaches.
This is an evolving field and Cochrane Qualitative Research Methods Group members are actively involved in contributing to knowledge and practice in this area. We, however, feel that it is important to consider and debate the arguments for and against critical appraisal in qualitative evidence synthesis.
Over one hundred tools and frameworks are available to aid the appraisal of qualitative research, mirroring those available for the appraisal of methodological quality in randomized trials and other forms of quantitative research (Vermeire 2002, Cote 2005). However, it is important to recognize that questions about ‘quality’ are very different in the context of qualitative research. Formal appraisal processes and standards of evidence presented as rigid checklists informing an ‘in or out’ decision can be argued to be inappropriate for qualitative research (Popay 1998a, Barbour 2001, Spencer 2003). Rather, such tools are perhaps best utilized as part of a process of exploration and interpretation. Studies rated of low methodological quality on the basis of a rigid formulaic method can generate new insights, grounded in the data, while methodologically sound studies may suffer from poor interpretation, leading to insufficient insight into the phenomenon under study. Dixon-Woods et al. compared three structured appraisal approaches and concluded that structured approaches may not produce greater consistency of judgements about whether to include qualitative papers in a systematic review (Dixon-Woods 2007).
A further issue relates to the timing of quality assessment and when outcomes from the process should be taken into account – should critical appraisal be viewed as a hurdle for establishing a quality threshold or as a filter for mediating the differing strength of the resultant messages from included research?
If authors decide to incorporate quality appraisal as part of the systematic review process then they may use the framework that is integral to the particular method (such as the Evidence for Policy and Practice Information (EPPI) approach or Joanna Briggs Institute (JBI) approach), or select any published qualitative appraisal tool, framework or checklist. Spencer et al. have undertaken a review of many of the current appraisal frameworks and checklists, which authors may find helpful in deciding which approach to apply (Spencer 2003). Expert judgement is also an important factor when appraising the quality of studies.
Key references reflecting this debate are included in Section 20.6.6: Further Reading.