The next stage of research is to undertake critical appraisal of the methodological quality of the remaining health economics studies, in order to address risk of bias. Variability in the quality of the conduct and reporting in health economic analyses is well documented (Neumann 2005). The core objective of critical appraisal of health economics studies is to assess whether they describe methods, assumptions, models and possible biases in a way that is transparent and fully supported by available evidence, the strength of which is made easily accessible to any critical reader (Rennie 2000).
Critical appraisal of health economics studies can be informed by the use of checklists that have been developed to guide assessments of methodological quality. Where checklists are used to inform critical appraisal of health economics studies in a Cochrane review, bibliographic details of the checklist should be cited in the ‘Data collection and analysis’ section. Whichever checklists are used, it is also useful to consider including additional tables to summarize completed checklists for included health economics studies in the published review.
The reliability of a full economic evaluation (see Section 15.5.2) is in part predicated on its use of reliable effectiveness data, so part of the critical appraisal of a full economic evaluation conducted alongside a single effectiveness study (e.g. a randomized trial) involves considering all those sources of potential bias that may apply to the effectiveness study used (see Chapter 8). For this type of full economic evaluation study, the critical appraisal will therefore consist of the following two parts:
Assessment of the risk of bias in results of the single effectiveness study on which the full economic evaluation study is based, informed by a recognized checklist for effectiveness studies.
Assessment of the methodological quality of the full economic evaluation study, informed by a recognized checklist for economic evaluations conducted alongside single study designs.
A number of checklists have been developed to guide critical appraisal of health economics studies. Whilst no checklists have been formally validated, two have received more scrutiny than most:
British Medical Journal Checklist for authors and peer reviewers of economic submissions (Drummond 1996);
CHEC list for assessment of methodological quality of economic evaluations (Evers 2005).
These checklists are reproduced in Figures 15.5.a and 15.5.b. Use of the ‘Drummond checklist’ and the ‘Evers checklist’ is recommended in Cochrane reviews to inform appraisal of the methodological quality of full economic evaluations conducted alongside single effectiveness studies, and also to inform critical appraisal of partial economic evaluations using the subset of applicable checklist items (see also Section 15.1.2).
If the scope of the critical review of health economics studies encompasses relevant economic modelling studies (see Section 15.2.3), then assessments of the methodological quality of such studies will need to be informed by a different checklist, since the ‘Drummond checklist’ and ‘Evers checklist’ are relevant but not sufficient for modelling studies. The ‘Phillips checklist’ is recommended to inform critical appraisal of the methodological quality of economic modelling studies (Philips 2004). Use of this checklist can be supplemented by referring to a published hierarchy of data sources which sets out the sources of data that are recognized as the best available sources to inform each parameter in an economic model (Cooper 2005).
Critical appraisal of the methodological quality of all types of full economic evaluation can usefully be informed by a corresponding NHS EED structured abstract, if available, to supplement the use of checklists (see also Section 15.3.2). This is because NHS EED structured abstracts include critical appraisal of study quality based on the same dimensions of quality reflected in the checklists recommended above.
There are as yet no widely validated minimum methodological criteria to be applied to screening economic studies for inclusion in systematic reviews. Decisions to include or exclude such studies will therefore need to be made on the basis of an overall judgement regarding their methodological quality, as well as their relevance in terms of the economic questions, interventions, populations and outcomes being studied (see Section 15.4.1). Eligibility criteria relating to dimensions of the methodological quality of health economics studies should be stated in the ‘Data collection and analysis’ section.
It is also important to highlight that, to date, there has been relatively little empirical research to investigate the impact upon the results of a critical review of health economics studies, of decisions to include economic studies that meet some but not all standards of methodological quality. However, as with choice of eligibility criteria relating to quality and design of effectiveness studies, and to the design of health economics studies (see also Section 15.2.3), it is plausible that use of different data sources for measures of resource use, cost and/or cost-effectiveness has at least the potential to impact on results (see also Section 15.7).