Assessing the quality of public health and health promotion studies, and their resulting risk of bias, may be difficult, partly due to the wide variety of study designs used. Authors need to consider the criteria to be used to assess quality at the planning stage of the review. Appraisal criteria will depend on the type of study included in the review. Authors should be guided by the Cochrane Review Group (CRG) editing their review and the appraisal tools they use. However the following describes tools which may be useful for assessing studies of public health and health promotion interventions.
The risk of bias in randomized trials should be assessed using the Collaboration’s ‘Risk of bias’ tool described in Chapter 8 (Section 8.5).
Issues for cluster-randomized trials are discussed in Chapter 16 (Section 16.3.2).
For risk of bias in non-randomized studies, authors should consult Chapter 13 (Section 13.5).
Authors may choose to use the Quality Assessment Tool for Quantitative Studies (Effective Public Health Practice Project 2007). This tool was developed by the Effective Public Health Practice Project, Canada, and covers any quantitative study design. The tool takes between 10-15 minutes to complete. A comprehensive dictionary for the assessment tool is also published on their web site (www.myhamilton.ca/myhamilton/cityandgovernment/healthandsocialservices/research/ephpp/ephpp.htm). This tool includes components of intervention integrity and was judged to be suitable to use in systematic reviews of effectiveness in the review by Deeks et al. (Deeks 2003).
Guidance is available from the Cochrane Effective Practice and Organisation of Care Group on interrupted time series and controlled before-and-after studies (Cochrane EPOC Group 2008).
The results of uncontrolled studies (also called before-and-after studies without a control group) should be treated with caution. The absence of a comparison group makes it impossible to know what would have happened without the intervention. Some of the particular problems with interpreting data from uncontrolled studies include susceptibility to problems with confounding (including seasonality) and regression to the mean.