Discrepancy between mega-trial and small studies in meta-analysis of rare events_Chinese Journal of Evidence-Based Medicine

Authors：

YU Tianqi , XU Wentao , YOU Yueyuan ,  LI Jing

Research Center of Evidence Based Medicine and Clinical Epidemiology, West China Hospital, Sichuan University, Chengdu 610041, P.R.China;

Corresponding author：

LI Jing, Email: lijing68@hotmail.com

Keywords：

Large study; Small study; Meta-analysis; Rare event; Cochrane systematic review

DOI：

10.7507/1672-2531.202109115

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective To explore the differences between large and small studies in rare events meta-analysis. Methods Empirical data were collected from The Cochrane Systematic Review Database from January 2003 to May 2018. Meta-analyses with rare events, binary outcomes involving at least 5 studies, and at least 1 large study were screened. Peto and classical ORs were used to compare the magnitude, direction and P-value. Results A total of 214 meta-analyses were included. Among 214 pairs of ORs of large and small studies, 66 pairs (30.84%) were inconsistent in the direction of ORs based on Peto OR (Kappa =0.33), and 69 pairs (32.24%) were inconsistent in the direction of ORs based on classical OR. The Peto ORs resulted in smaller P-values compared to classic ORs in a substantial (83.18%) number of cases. Conclusion There are considerable differences between large and small studies in the results of meta-analysis of rare events.

Citation： YU Tianqi, XU Wentao, YOU Yueyuan, LI Jing. Discrepancy between mega-trial and small studies in meta-analysis of rare events. Chinese Journal of Evidence-Based Medicine, 2022, 22(1): 97-102. doi: 10.7507/1672-2531.202109115 Copy

1.	Guyatt GH, Oxman AD, Kunz R, et al. What is "quality of evidence" and why is it important to clinicians. BMJ, 2008, 336(7651): 995-998.
2.	Sutton AJ, Higgins JP. Recent developments in meta-analysis. Stat Med, 2008, 27(5): 625-650.
3.	Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol, 2011, 64(4): 383-394.
4.	Shadish WR, Lecy JD. The meta-analytic big bang. Res Synth Methods, 2015, 6(3): 246-264.
5.	Siddaway AP, Wood AM, Hedges LV. How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annu Rev Psychol, 2019, 70: 747-770.
6.	Borenstein M, Hedges LV, Higgins JP, et al. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods, 2010, 1(2): 97-111.
7.	Doi SA, Barendregt JJ, Khan S, et al. Advances in the meta-analysis of heterogeneous clinical trials I: the inverse variance heterogeneity model. Contemp Clin Trials, 2015, 45(Pt A): 130-138.
8.	Doi SA, Barendregt JJ, Khan S, et al. Advances in the meta-analysis of heterogeneous clinical trials II: the quality effects model. Contemp Clin Trials, 2015, 45(Pt A): 123-129.
9.	Villar J, Carroli G, Belizán JM. Predictive ability of meta-analyses of randomised controlled trials. Lancet, 1995, 345(8952): 772-776.
10.	LeLorier J, Grégoire G, Benhaddad A, et al. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med, 1997, 337(8): 536-542.
11.	Bhaumik DK, Amatya A, Normand SL, et al. Meta-analysis of rare binary adverse event data. J Am Stat Assoc, 2012, 107(498): 555-567.
12.	Pateras K, Nikolakopoulos S, Roes KCB. Prior distributions for variance parameters in a sparse-event meta-analysis of a few small trials. Pharm Stat, 2021, 20(1): 39-54.
13.	Mathes T, Kuss O. A comparison of methods for meta-analysis of a small number of studies with binary outcomes. Res Synth Methods, 2018, 9(3): 366-381.
14.	Kuss O. Statistical methods for meta-analyses including information from studies without any events-add nothing to nothing and succeed nevertheless. Stat Med, 2015, 34(7): 1097-1116.
15.	Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing. Use and avoidance of continuity corrections in meta-analysis of sparse data. Stat Med, 2004, 23(9): 1351-1375.
16.	Günhan BK, Röver C, Friede T. Random-effects meta-analysis of few studies involving rare events. Res Synth Methods, 2020, 11(1): 74-90.
17.	Lin L, Chu H, Murad MH, et al. Empirical comparison of publication bias tests in meta-analysis. J Gen Intern Med, 2018, 33(8): 1260-1267.
18.	Turner RM, Davey J, Clarke MJ, et al. Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane database of systematic reviews. Int J Epidemiol, 2012, 41(3): 818-827.
19.	Spineli LM, Pandis N, Salanti G. Reporting and handling missing outcome data in mental health: a systematic review of Cochrane systematic reviews and meta-analyses. Res Synth Methods, 2015, 6(2): 175-187.
20.	Kicinski M, Springate DA, Kontopantelis E. Publication bias in meta-analyses from the Cochrane database of systematic reviews. Stat Med, 2015, 34(20): 2781-2793.
21.	Lang DT. R as a Web Client–the RCurl package, 2007. Available at: http://www.omegahat.net/RCurl/RCurlJSS.pdf.
22.	Liu D, Liu RY, Xie M. Exact meta-analysis approach for discrete data and its application to 2 × 2 tables with rare events. J Am Stat Assoc, 2014, 109(508): 1450-1465.
23.	Jackson D, Law M, Stijnen T, et al. A comparison of seven random-effects models for meta-analyses that estimate the summary odds ratio. Stat Med, 2018, 37(7): 1059-1085.
24.	Shrier I, Platt RW, Steele RJ. Mega-trials vs. meta-analysis: precision vs. heterogeneity? Contemp Clin Trials, 2007, 28(3): 324-328.
25.	Yusuf S, Peto R, Lewis J, et al. Beta blockade during and after myocardial infarction: an overview of the randomized trials. Prog Cardiovasc Dis, 1985, 27(5): 335-371.
26.	Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1. 0, 2011. Available at: www. handbook. cochrane. org.
27.	Xu C, Li L, Lin L, et al. Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions. J Clin Epidemiol, 2020, 123: 91-99.
28.	Doi SAR, Furuya-Kanamori L, Thalib L, et al. Meta-analysis in evidence-based healthcare: a paradigm shift away from random effects is overdue. Int J Evid Based Healthc, 2017, 15(4): 152-160.
29.	Altman DG, Bland JM. Measurement in medicine: the analysis of method comparison studies. Statistician, 1983, 32(3): 307-317.
30.	McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb), 2012, 22(3): 276-282.
31.	Flather MD, Farkouh ME, Pogue JM, et al. Strengths and limitations of meta-analysis: larger studies may be more reliable. Control Clin Trials, 1997, 18(6): 568-579.
32.	Cappelleri JC, Ioannidis JP, Schmid CH, et al. Large trials vs meta-analysis of smaller trials: how do their results compare. JAMA, 1996, 276(16): 1332-1338.
33.	Rücker G, Carpenter JR, Schwarzer G. Detecting and adjusting for small-study effects in meta-analysis. Biom J, 2011, 53(2): 351-368.
34.	Lin L, Chu H. Quantifying publication bias in meta-analysis. Biometrics, 2018, 74(3): 785-794.
35.	Furuya-Kanamori L, Barendregt JJ, Doi SAR. A new improved graphical and quantitative method for detecting bias in meta-analysis. Int J Evid Based Healthc, 2018, 16(4): 195-203.
36.	Nüesch E, Trelle S, Reichenbach S, et al. Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study. BMJ, 2010, 341: c3515.
37.	Doi SA, Thalib L. A quality-effects model for meta-analysis. Epidemiology, 2008, 19(1): 94-100.
38.	Al Khalaf MM, Thalib L, Doi SA. Combining heterogenous studies using the random-effects model is a mistake and leads to inconclusive meta-analyses. J Clin Epidemiol, 2011, 64(2): 119-123.

1. Guyatt GH, Oxman AD, Kunz R, et al. What is "quality of evidence" and why is it important to clinicians. BMJ, 2008, 336(7651): 995-998.
2. Sutton AJ, Higgins JP. Recent developments in meta-analysis. Stat Med, 2008, 27(5): 625-650.
3. Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol, 2011, 64(4): 383-394.
4. Shadish WR, Lecy JD. The meta-analytic big bang. Res Synth Methods, 2015, 6(3): 246-264.
5. Siddaway AP, Wood AM, Hedges LV. How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annu Rev Psychol, 2019, 70: 747-770.
6. Borenstein M, Hedges LV, Higgins JP, et al. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods, 2010, 1(2): 97-111.
7. Doi SA, Barendregt JJ, Khan S, et al. Advances in the meta-analysis of heterogeneous clinical trials I: the inverse variance heterogeneity model. Contemp Clin Trials, 2015, 45(Pt A): 130-138.
8. Doi SA, Barendregt JJ, Khan S, et al. Advances in the meta-analysis of heterogeneous clinical trials II: the quality effects model. Contemp Clin Trials, 2015, 45(Pt A): 123-129.
9. Villar J, Carroli G, Belizán JM. Predictive ability of meta-analyses of randomised controlled trials. Lancet, 1995, 345(8952): 772-776.
10. LeLorier J, Grégoire G, Benhaddad A, et al. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med, 1997, 337(8): 536-542.
11. Bhaumik DK, Amatya A, Normand SL, et al. Meta-analysis of rare binary adverse event data. J Am Stat Assoc, 2012, 107(498): 555-567.
12. Pateras K, Nikolakopoulos S, Roes KCB. Prior distributions for variance parameters in a sparse-event meta-analysis of a few small trials. Pharm Stat, 2021, 20(1): 39-54.
13. Mathes T, Kuss O. A comparison of methods for meta-analysis of a small number of studies with binary outcomes. Res Synth Methods, 2018, 9(3): 366-381.
14. Kuss O. Statistical methods for meta-analyses including information from studies without any events-add nothing to nothing and succeed nevertheless. Stat Med, 2015, 34(7): 1097-1116.
15. Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing. Use and avoidance of continuity corrections in meta-analysis of sparse data. Stat Med, 2004, 23(9): 1351-1375.
16. Günhan BK, Röver C, Friede T. Random-effects meta-analysis of few studies involving rare events. Res Synth Methods, 2020, 11(1): 74-90.
17. Lin L, Chu H, Murad MH, et al. Empirical comparison of publication bias tests in meta-analysis. J Gen Intern Med, 2018, 33(8): 1260-1267.
18. Turner RM, Davey J, Clarke MJ, et al. Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane database of systematic reviews. Int J Epidemiol, 2012, 41(3): 818-827.
19. Spineli LM, Pandis N, Salanti G. Reporting and handling missing outcome data in mental health: a systematic review of Cochrane systematic reviews and meta-analyses. Res Synth Methods, 2015, 6(2): 175-187.
20. Kicinski M, Springate DA, Kontopantelis E. Publication bias in meta-analyses from the Cochrane database of systematic reviews. Stat Med, 2015, 34(20): 2781-2793.
21. Lang DT. R as a Web Client–the RCurl package, 2007. Available at: http://www.omegahat.net/RCurl/RCurlJSS.pdf.
22. Liu D, Liu RY, Xie M. Exact meta-analysis approach for discrete data and its application to 2 × 2 tables with rare events. J Am Stat Assoc, 2014, 109(508): 1450-1465.
23. Jackson D, Law M, Stijnen T, et al. A comparison of seven random-effects models for meta-analyses that estimate the summary odds ratio. Stat Med, 2018, 37(7): 1059-1085.
24. Shrier I, Platt RW, Steele RJ. Mega-trials vs. meta-analysis: precision vs. heterogeneity? Contemp Clin Trials, 2007, 28(3): 324-328.
25. Yusuf S, Peto R, Lewis J, et al. Beta blockade during and after myocardial infarction: an overview of the randomized trials. Prog Cardiovasc Dis, 1985, 27(5): 335-371.
26. Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1. 0, 2011. Available at: www. handbook. cochrane. org.
27. Xu C, Li L, Lin L, et al. Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions. J Clin Epidemiol, 2020, 123: 91-99.
28. Doi SAR, Furuya-Kanamori L, Thalib L, et al. Meta-analysis in evidence-based healthcare: a paradigm shift away from random effects is overdue. Int J Evid Based Healthc, 2017, 15(4): 152-160.
29. Altman DG, Bland JM. Measurement in medicine: the analysis of method comparison studies. Statistician, 1983, 32(3): 307-317.
30. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb), 2012, 22(3): 276-282.
31. Flather MD, Farkouh ME, Pogue JM, et al. Strengths and limitations of meta-analysis: larger studies may be more reliable. Control Clin Trials, 1997, 18(6): 568-579.
32. Cappelleri JC, Ioannidis JP, Schmid CH, et al. Large trials vs meta-analysis of smaller trials: how do their results compare. JAMA, 1996, 276(16): 1332-1338.
33. Rücker G, Carpenter JR, Schwarzer G. Detecting and adjusting for small-study effects in meta-analysis. Biom J, 2011, 53(2): 351-368.
34. Lin L, Chu H. Quantifying publication bias in meta-analysis. Biometrics, 2018, 74(3): 785-794.
35. Furuya-Kanamori L, Barendregt JJ, Doi SAR. A new improved graphical and quantitative method for detecting bias in meta-analysis. Int J Evid Based Healthc, 2018, 16(4): 195-203.
36. Nüesch E, Trelle S, Reichenbach S, et al. Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study. BMJ, 2010, 341: c3515.
37. Doi SA, Thalib L. A quality-effects model for meta-analysis. Epidemiology, 2008, 19(1): 94-100.
38. Al Khalaf MM, Thalib L, Doi SA. Combining heterogenous studies using the random-effects model is a mistake and leads to inconclusive meta-analyses. J Clin Epidemiol, 2011, 64(2): 119-123.

Previous Article
Reporting quality and its influencing factors of literature screening results for systematic reviews on acupuncture

Chinese Journal of Evidence-Based Medicine

Discrepancy between mega-trial and small studies in meta-analysis of rare events

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Format

Content