Methods and processes for producing a systematic review of predictive model studies_Chinese Journal of Evidence-Based Medicine

Authors：

LU Xiaodan ¹ , WEI Jianhua ² ,  SHEN Jiantong ¹ , HAN Hui ³ , ZHU Yujia ³ , CHEN Yaping ⁴ , HUA Yuting ¹ , LIU Yujin ¹ , ZHENG Lei ¹ , MO Hangfeng ¹ , CAI Tingting ¹ , XIA Yunhui ¹

1. School of Medicine, Huzhou University, Huzhou 313000, P. R. China;
2. The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou 310003, P. R. China;
3. The First Affiliated Hospital of Huzhou University, Huzhou 313000, P. R. China;
4. Huzhou Wuxing District Daxi Hospital, Huzhou 313000, P. R. China;

Corresponding author：

SHEN Jiantong, Email: sjiantong@163.com

Keywords：

Prediction model; Systematic review; Methodology; Evidence-based medicine

DOI：

10.7507/1672-2531.202212112

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

As precision medicine continues to gain momentum, the number of predictive model studies is increasing. However, the quality of the methodology and reporting varies greatly, which limits the promotion and application of these models in clinical practice. Systematic reviews of prediction models draw conclusions by summarizing and evaluating the performance of such models in different settings and populations, thus promoting their application in practice. Although the number of systematic reviews of predictive model studies has increased in recent years, the methods used are still not standardized and the quality varies greatly. In this paper, we combine the latest advances in methodologies both domestically and abroad, and summarize the production methods and processes of a systematic review of prediction models. The aim of this study is to provide references for domestic scholars to produce systematic reviews of prediction models.

Citation： LU Xiaodan, WEI Jianhua, SHEN Jiantong, HAN Hui, ZHU Yujia, CHEN Yaping, HUA Yuting, LIU Yujin, ZHENG Lei, MO Hangfeng, CAI Tingting, XIA Yunhui. Methods and processes for producing a systematic review of predictive model studies. Chinese Journal of Evidence-Based Medicine, 2023, 23(5): 602-609. doi: 10.7507/1672-2531.202212112 Copy

1.	Faulkner E, Holtorf AP, Walton S, et al. Being precise about precision medicine: what should value frameworks incorporate to address precision medicine. A report of the personalized precision medicine special interest group. Value Health, 2020, 23(5): 529-539.
2.	Wynants L, Van Calster B, Collins GS, et al. Update to living systematic review on prediction models for diagnosis and prognosis of covid-19. BMJ, 2021, 372: n236.
3.	Damen JA, Hooft L, Schuit E, et al. Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ, 2016, 353: i2416.
4.	Bellou V, Belbasis L, Konstantinidis AK, et al. Prognostic models for outcome prediction in patients with chronic obstructive pulmonary disease: systematic review and critical appraisal. BMJ, 2019, 367: l5358.
5.	Kreuzberger N, Damen JA, Trivella M, et al. Prognostic models for newly-diagnosed chronic lymphocytic leukaemia in adults: a systematic review and meta-analysis. Cochrane Database Syst Rev, 2020, (7): CD012022.
6.	Black N, Henderson I, Al Wattar BH, et al. Predictive models for estimating the probability of successful vaginal birth after cesarean delivery: a systematic review. Obstet Gynecol, 2022, 140(5): 821-841.
7.	Bouwmeester W, Zuithoff NP, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med, 2012, 9(5): 1-12.
8.	Debray TP, Damen JA, Snell KI, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ, 2017, 356: i6460.
9.	Heus P, Damen JAAG, Pajouheshnia R, et al. Poor reporting of multivariable prediction model studies: towards a targeted implementation strategy of the TRIPOD statement. BMC Med, 2018, 16(1): 120.
10.	Damen JAA, Moons KGM, van Smeden M, et al. How to conduct a systematic review and meta-analysis of prognostic model studies. Clin Microbiol Infect, 2023, 29(4): 434-440.
11.	Belbasis L, Panagiotou OA. Reproducibility of prediction models in health services research. BMC Res Notes, 2022, 15(1): 204.
12.	Moons KG, Hooft L, Williams K, et al. Implementing systematic reviews of prognosis studies in Cochrane. Cochrane Database Syst Rev, 2018, 10: ED000129.
13.	Moons KG, de Groot JA, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med, 2014, 11(10): e1001744.
14.	Geersing GJ, Bouwmeester W, Zuithoff P, et al. Search filters for finding prognostic and diagnostic prediction studies in Medline to enhance systematic reviews. PLoS One, 2012, 7(2): e32844.
15.	Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc, 2001, 8(4): 391-397.
16.	Wong SS, Wilczynski NL, Haynes RB, et al. Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE. AMIA Annu Symp Proc, 2003, 2003: 728-732.
17.	Bonnett LJ, Snell KIE, Collins GS, et al. Guide to presenting clinical prediction models for use in clinical settings. BMJ, 2019, 365: l737.
18.	Van Calster B, Wynants L, Verbeek JFM, et al. Reporting and interpreting decision curve analysis: a guide for investigators. Eur Urol, 2018, 74(6): 796-804.
19.	Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ, 2016, 352: i6.
20.	Alba AC, Agoritsas T, Walsh M, et al. Discrimination and calibration of clinical prediction models: users' guides to the medical literature. JAMA, 2017, 318(14): 1377-1384.
21.	Debray TP, Vergouwe Y, Koffijberg H, et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol, 2015, 68(3): 279-289.
22.	Vergouwe Y, Moons KG, Steyerberg EW. External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol, 2010, 172(8): 971-980.
23.	Van Calster B, McLernon DJ, van Smeden M, et al. Calibration: the Achilles heel of predictive analytics. BMC Med, 2019, 17(1): 230.
24.	Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med, 2019, 170(1): 51-58.
25.	陈香萍, 张奕, 庄一渝, 等. PROBAST: 诊断或预后多因素预测模型研究偏倚风险的评估工具. 中国循证医学杂志, 2020, 20(6): 737-744.
26.	Ban JW, Emparanza JI, Urreta I, et al. Design characteristics influence performance of clinical prediction rules in validation: a meta-epidemiological study. PLoS One, 2016, 11(1): e0145779.
27.	Debray TP, Damen JA, Riley RD, et al. A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes. Stat Methods Med Res, 2019, 28(9): 2768-2786.
28.	van Klaveren D, Steyerberg EW, Perel P, et al. Assessing discriminative ability of risk models in clustered data. BMC Med Res Methodol, 2014, 14: 5.
29.	IntHout J, Ioannidis JP, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol, 2014, 14: 25.
30.	Riley RD, Higgins JP, Deeks JJ. Interpretation of random effects meta-analyses. BMJ, 2011, 342: d549.
31.	Higgins JP, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc, 2009, 172(1): 137-159.
32.	Pennells L, Kaptoge S, White IR, et al. Assessing risk prediction models using individual participant data from multiple studies. Am J Epidemiol, 2014, 179(5): 621-632.
33.	Riley RD, Ensor J, Snell KI, et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ, 2016, 353: i3140.
34.	Belias M, Rovers MM, Reitsma JB, et al. Statistical approaches to identify subgroups in meta-analysis of individual participant data: a simulation study. BMC Med Res Methodol, 2019, 19(1): 183.
35.	Snell KI, Hua H, Debray TP, et al. Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model. J Clin Epidemiol, 2016, 69: 40-50.
36.	Riley RD, Jackson D, Salanti G, et al. Multivariate and network meta-analysis of multiple outcomes and multiple treatments: rationale, concepts, and examples. BMJ, 2017, 358: j3932.
37.	Wynants L, Riley RD, Timmerman D, et al. Random-effects meta-analysis of the clinical utility of tests and prediction models. Stat Med, 2018, 37(12): 2034-2052.
38.	Iorio A, Spencer FA, Falavigna M, et al. Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. BMJ, 2015, 350: h870.
39.	Foroutan F, Guyatt G, Zuk V, et al. GRADE Guidelines 28: use of GRADE for the assessment of evidence about prognostic factors: rating certainty in identification of groups of patients with different absolute risks. J Clin Epidemiol, 2020, 121: 62-70.
40.	Huguet A, Hayden JA, Stinson J, et al. Judging the quality of evidence in reviews of prognostic factor research: adapting the GRADE framework. Syst Rev, 2013, 2: 71.
41.	Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 2021, 372: n71.
42.	Page MJ, Moher D, Bossuyt PM, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ, 2021, 372: n160.
43.	Stewart LA, Clarke M, Rovers M, et al. Preferred reporting items for systematic review and meta-analyses of individual participant data: the PRISMA-IPD Statement. JAMA, 2015, 313(16): 1657-1665.
44.	Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ, 2015, 350: g7594.
45.	Debray TP, Riley RD, Rovers MM, et al. Individual participant data (IPD) meta-analyses of diagnostic and prognostic modeling studies: guidance on their use. PLoS Med, 2015, 12(10): e1001886.
46.	Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med, 2015, 162(1): W1-73.
47.	Collins GS, Dhiman P, Andaur Navarro CL, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open, 2021, 11(7): e048008.
48.	Collins GS, Moons KGM. Reporting of artificial intelligence prediction models. Lancet, 2019, 393(10181): 1577-1579.

1. Faulkner E, Holtorf AP, Walton S, et al. Being precise about precision medicine: what should value frameworks incorporate to address precision medicine. A report of the personalized precision medicine special interest group. Value Health, 2020, 23(5): 529-539.
2. Wynants L, Van Calster B, Collins GS, et al. Update to living systematic review on prediction models for diagnosis and prognosis of covid-19. BMJ, 2021, 372: n236.
3. Damen JA, Hooft L, Schuit E, et al. Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ, 2016, 353: i2416.
4. Bellou V, Belbasis L, Konstantinidis AK, et al. Prognostic models for outcome prediction in patients with chronic obstructive pulmonary disease: systematic review and critical appraisal. BMJ, 2019, 367: l5358.
5. Kreuzberger N, Damen JA, Trivella M, et al. Prognostic models for newly-diagnosed chronic lymphocytic leukaemia in adults: a systematic review and meta-analysis. Cochrane Database Syst Rev, 2020, (7): CD012022.
6. Black N, Henderson I, Al Wattar BH, et al. Predictive models for estimating the probability of successful vaginal birth after cesarean delivery: a systematic review. Obstet Gynecol, 2022, 140(5): 821-841.
7. Bouwmeester W, Zuithoff NP, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med, 2012, 9(5): 1-12.
8. Debray TP, Damen JA, Snell KI, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ, 2017, 356: i6460.
9. Heus P, Damen JAAG, Pajouheshnia R, et al. Poor reporting of multivariable prediction model studies: towards a targeted implementation strategy of the TRIPOD statement. BMC Med, 2018, 16(1): 120.
10. Damen JAA, Moons KGM, van Smeden M, et al. How to conduct a systematic review and meta-analysis of prognostic model studies. Clin Microbiol Infect, 2023, 29(4): 434-440.
11. Belbasis L, Panagiotou OA. Reproducibility of prediction models in health services research. BMC Res Notes, 2022, 15(1): 204.
12. Moons KG, Hooft L, Williams K, et al. Implementing systematic reviews of prognosis studies in Cochrane. Cochrane Database Syst Rev, 2018, 10: ED000129.
13. Moons KG, de Groot JA, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med, 2014, 11(10): e1001744.
14. Geersing GJ, Bouwmeester W, Zuithoff P, et al. Search filters for finding prognostic and diagnostic prediction studies in Medline to enhance systematic reviews. PLoS One, 2012, 7(2): e32844.
15. Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc, 2001, 8(4): 391-397.
16. Wong SS, Wilczynski NL, Haynes RB, et al. Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE. AMIA Annu Symp Proc, 2003, 2003: 728-732.
17. Bonnett LJ, Snell KIE, Collins GS, et al. Guide to presenting clinical prediction models for use in clinical settings. BMJ, 2019, 365: l737.
18. Van Calster B, Wynants L, Verbeek JFM, et al. Reporting and interpreting decision curve analysis: a guide for investigators. Eur Urol, 2018, 74(6): 796-804.
19. Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ, 2016, 352: i6.
20. Alba AC, Agoritsas T, Walsh M, et al. Discrimination and calibration of clinical prediction models: users' guides to the medical literature. JAMA, 2017, 318(14): 1377-1384.
21. Debray TP, Vergouwe Y, Koffijberg H, et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol, 2015, 68(3): 279-289.
22. Vergouwe Y, Moons KG, Steyerberg EW. External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol, 2010, 172(8): 971-980.
23. Van Calster B, McLernon DJ, van Smeden M, et al. Calibration: the Achilles heel of predictive analytics. BMC Med, 2019, 17(1): 230.
24. Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med, 2019, 170(1): 51-58.
25. 陈香萍, 张奕, 庄一渝, 等. PROBAST: 诊断或预后多因素预测模型研究偏倚风险的评估工具. 中国循证医学杂志, 2020, 20(6): 737-744.
26. Ban JW, Emparanza JI, Urreta I, et al. Design characteristics influence performance of clinical prediction rules in validation: a meta-epidemiological study. PLoS One, 2016, 11(1): e0145779.
27. Debray TP, Damen JA, Riley RD, et al. A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes. Stat Methods Med Res, 2019, 28(9): 2768-2786.
28. van Klaveren D, Steyerberg EW, Perel P, et al. Assessing discriminative ability of risk models in clustered data. BMC Med Res Methodol, 2014, 14: 5.
29. IntHout J, Ioannidis JP, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol, 2014, 14: 25.
30. Riley RD, Higgins JP, Deeks JJ. Interpretation of random effects meta-analyses. BMJ, 2011, 342: d549.
31. Higgins JP, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc, 2009, 172(1): 137-159.
32. Pennells L, Kaptoge S, White IR, et al. Assessing risk prediction models using individual participant data from multiple studies. Am J Epidemiol, 2014, 179(5): 621-632.
33. Riley RD, Ensor J, Snell KI, et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ, 2016, 353: i3140.
34. Belias M, Rovers MM, Reitsma JB, et al. Statistical approaches to identify subgroups in meta-analysis of individual participant data: a simulation study. BMC Med Res Methodol, 2019, 19(1): 183.
35. Snell KI, Hua H, Debray TP, et al. Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model. J Clin Epidemiol, 2016, 69: 40-50.
36. Riley RD, Jackson D, Salanti G, et al. Multivariate and network meta-analysis of multiple outcomes and multiple treatments: rationale, concepts, and examples. BMJ, 2017, 358: j3932.
37. Wynants L, Riley RD, Timmerman D, et al. Random-effects meta-analysis of the clinical utility of tests and prediction models. Stat Med, 2018, 37(12): 2034-2052.
38. Iorio A, Spencer FA, Falavigna M, et al. Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. BMJ, 2015, 350: h870.
39. Foroutan F, Guyatt G, Zuk V, et al. GRADE Guidelines 28: use of GRADE for the assessment of evidence about prognostic factors: rating certainty in identification of groups of patients with different absolute risks. J Clin Epidemiol, 2020, 121: 62-70.
40. Huguet A, Hayden JA, Stinson J, et al. Judging the quality of evidence in reviews of prognostic factor research: adapting the GRADE framework. Syst Rev, 2013, 2: 71.
41. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 2021, 372: n71.
42. Page MJ, Moher D, Bossuyt PM, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ, 2021, 372: n160.
43. Stewart LA, Clarke M, Rovers M, et al. Preferred reporting items for systematic review and meta-analyses of individual participant data: the PRISMA-IPD Statement. JAMA, 2015, 313(16): 1657-1665.
44. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ, 2015, 350: g7594.
45. Debray TP, Riley RD, Rovers MM, et al. Individual participant data (IPD) meta-analyses of diagnostic and prognostic modeling studies: guidance on their use. PLoS Med, 2015, 12(10): e1001886.
46. Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med, 2015, 162(1): W1-73.
47. Collins GS, Dhiman P, Andaur Navarro CL, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open, 2021, 11(7): e048008.
48. Collins GS, Moons KGM. Reporting of artificial intelligence prediction models. Lancet, 2019, 393(10181): 1577-1579.

Chinese Journal of Evidence-Based Medicine

Methods and processes for producing a systematic review of predictive model studies

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content