10.4.4.6  Testing for excess of studies with significant results

Ioannidis and Trikalinos propose a simple test that aims to evaluate whether there is an excess of studies that have formally statistically significant results (Ioannidis 2007a). The test compares the number of studies that have formally statistically significant results with the number of statistically significant results expected under different assumptions about the magnitude of the effect size.  The simplest assumption is that the effect size is equal to the observed summary effect in the meta-analysis (but this may introduce an element of circularity).  Other values for the underlying effect size, and different thresholds of significance, may be used. Hence, like the contour funnel plots described in Section 10.4.1, but unlike the regression tests, this method considers the distribution of the significance of study results. However, unlike either the regression tests or contour funnel plots, the test does not make any assumption about small-study effects. An excess of significant results can reflect either suppression of whole studies or related selective/manipulative analysis and reporting practices that would cause similar excess.

 

The test has limited power, as do most other tests, when there are few studies and when there are few studies with significant results. Because the test has not been rigorously evaluated through simulation in comparison with alternative tests and under different scenarios, we currently do not recommend the test as an alternative to those described in Section 10.4.3 .

 

A novel feature of the test is that it can be applied across a large number of meta-analyses on the same research field to examine the extent of publication and selective reporting biases across a whole domain of clinical research. Again, further evaluation of this approach would be welcome.