Interpretation of the step-by-step guide for developing clinical prediction models in 2024_Chinese Journal of Evidence-Based Medicine

Authors：

SHUAI Yuxing ¹ , SHEN Xuewei ² , WANG Jingyi ³ , CHEN Zhuoqun ¹ , FENG Taichuan ¹ , MA Qing ¹ , DU Liang ⁴ ,  CHEN Xinlin ¹

1. School of Basic Medical Sciences, Guangzhou University of Chinese Medicine, Guangzhou 510006, P. R. China;
2. School of Public Health and Management, Guangzhou University of Chinese Medicine, Guangzhou 510006, P. R. China;
3. The Fourth Clinical Medical College, Guangzhou University of Chinese Medicine, Guangzhou 510403, P. R. China;
4. West China Hospital, Sichuan University, Chengdu 610041, P. R. China;

Corresponding author：

CHEN Xinlin, Email: chenxlsums@126.com

Keywords：

Clinical prediction model; Construction and validation; Guide; Interpretation

DOI：

10.7507/1672-2531.202502092

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Clinical prediction models refer to models that can predict the probability of the occurrence of a certain clinical outcome event of the research objects, and they have important value in fields such as disease risk stratification, prognosis prediction, and precision medical decision - making. To further standardize this methodology, in 2024, an international multidisciplinary expert group composed of institutions from Switzerland, the Netherlands, the United Kingdom, and others, based on the TRIPOD statement and the PROBAST assessment tool, jointly released the "Step - by - step guide for developing clinical prediction models". This guide systematically constructs 13 steps: defining the objective, creating a team, conducting a literature review, developing a protocol, choosing to develop a new model or update an existing model, defining the outcome measure, identifying candidate predictors, collecting and checking data, determining the sample size, handling missing data, fitting the prediction model, evaluating the performance of the prediction model, determining the final model, performing decision curve analysis, evaluating the predictive ability of individual predictors, writing a report and publishing the results. This paper deeply analyzes the steps of this guide, aiming to provide a reference for clinical researchers.

1.	Dankers F, Traverso A, Wee L, et al. Prediction modeling methodology. Kubben P, Dumontier M, Dekker A. Fundamentals of clinical data science. Cham (CH): Springer Copyright, 2019.
2.	谷鸿秋, 周支瑞, 章仲恒, 等. 临床预测模型: 基本概念、应用场景及研究思路. 中国循证心血管医学杂志, 2018, 10(12): 1454-1456,1462.
3.	Lee YH, Bang H, Kim DJ. How to establish clinical prediction models. Endocrinol Metab (Seoul), 2016, 31(1): 38-44.
4.	Bouwmeester W, Zuithoff NP, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med, 2012, 9(5): 1-12.
5.	Efthimiou O, Seo M, Chalkou K, et al. Developing clinical prediction models: a step-by-step guide. BMJ, 2024, 386: e078276.
6.	Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med, 2015, 162(1): 55-63.
7.	Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med, 2019, 170(1): 51-58.
8.	Ramspek CL, Jager KJ, Dekker FW, et al. External validation of prognostic models: what, why, how, when and where. Clin Kidney J, 2020, 14(1): 49-58.
9.	Dankowski T, Ziegler A. Calibrating random forests for probability estimation. Stat Med, 2016, 35(22): 3949-3960.
10.	Schuurmans M, Saha A, Alves N, et al. End-to-end prognostication in pancreatic cancer by multimodal deep learning: a retrospective, multicenter study. Eur Radiol, 2025.
11.	Altman DG, Royston P. The cost of dichotomising continuous variables. BMJ, 2006, 332(7549): 1080.
12.	Fedorov V, Mannino F, Zhang R. Consequences of dichotomization. Pharm Stat, 2009, 8(1): 50-61.
13.	Leisman DE, Harhay MO, Lederer DJ, et al. Development and reporting of prediction models: guidance for authors from editors of respiratory, sleep, and critical care journals. Crit Care Med, 2020, 48(5): 623-633.
14.	Rhon DI, Teyhen DS, Collins GS, et al. Predictive models for musculoskeletal injury risk: why statistical approach makes all the difference. BMJ Open Sport Exerc Med, 2022, 8(4): e001388.
15.	Collins GS, Ogundimu EO, Cook JA, et al. Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model. Stat Med, 2016, 35(23): 4124-4135.
16.	Sanchez-Pinto LN, Venable LR, Fahrenbach J, et al. Comparison of variable selection methods for clinical predictive modeling. Int J Med Inform, 2018, 116: 10-17.
17.	Taipale H, Schneider-Thoma J, Pinzón-Espinosa J, et al. Representation and outcomes of individuals with schizophrenia seen in everyday practice who are ineligible for randomized clinical trials. JAMA Psychiatry, 2022, 79(3): 210-218.
18.	Bennett DA, Landry D, Little J, et al. Systematic review of statistical approaches to quantify, or correct for, measurement error in a continuous exposure in nutritional epidemiology. BMC Med Res Methodol, 2017, 17(1): 146.
19.	Yan F, Chen X, Quan X, et al. Association between the stress hyperglycemia ratio and 28-day all-cause mortality in critically ill patients with sepsis: a retrospective cohort study and predictive model establishment based on machine learning. Cardiovasc Diabetol, 2024, 23(1): 163.
20.	Barkley SE, Mason JL, Kappelman MM. An assessment of entering freshman medical students' knowledge of and attitudes toward AIDS. Res Med Educ, 1988, 27: 221-225.
21.	van der Ploeg T, Austin PC, Steyerberg EW. Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Med Res Methodol, 2014, 14: 137.
22.	Infante G, Miceli R, Ambrogi F. Sample size and predictive performance of machine learning methods with survival data: a simulation study. Stat Med, 2023, 42(30): 5657-5675.
23.	Sterne JA, White IR, Carlin JB, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ, 2009, 338: b2393.
24.	Sisk R, Sperrin M, Peek N, et al. Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: a simulation study. Stat Methods Med Res, 2023, 32(8): 1461-1477.
25.	Schafer JL. Multiple imputation: a primer. Stat Methods Med Res, 1999, 8(1): 3-15.
26.	Deo RC. Machine learning in medicine. Circulation, 2015, 132(20): 1920-1930.
27.	Lo Vercio L, Amador K, Bannister JJ, et al. Supervised machine learning tools: a tutorial for clinicians. J Neural Eng, 2020, 17(6): abbff2.
28.	Andaur Navarro CL, Damen JAA, Takada T, et al. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ, 2021, 375: n2281.
29.	Putter H, Fiocco M, Geskus RB. Tutorial in biostatistics: competing risks and multi-state models. Stat Med, 2007, 26(11): 2389-2430.
30.	Steyerberg EW, Eijkemans MJ, Habbema JD. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol, 1999, 52(10): 935-942.
31.	Van Calster B, van Smeden M, De Cock B, et al. Regression shrinkage methods for clinical prediction models do not guarantee improved performance: simulation study. Stat Methods Med Res, 2020, 29(11): 3166-3178.
32.	Riley RD, Snell KIE, Martin GP, et al. Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small. J Clin Epidemiol, 2021, 132: 88-96.
33.	Wood AM, Royston P, White IR. The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data. Biom J, 2015, 57(4): 614-632.
34.	Steyerberg EW. Validation in prediction research: the waste by data splitting. J Clin Epidemiol, 2018, 103: 131-133.
35.	Collins GS, Dhiman P, Ma J, et al. Evaluation of clinical prediction models (part 1): from development to external validation. BMJ, 2024, 384: e074819.
36.	Lynam AL, Dennis JM, Owen KR, et al. Logistic regression has similar performance to optimised machine learning algorithms in a clinical setting: application to the discrimination between type 1 and type 2 diabetes in young adults. Diagn Progn Res, 2020, 4: 6.
37.	National Institute for Health and Care Excellence: Guidelines. Cardiovascular disease: risk assessment and reduction, including lipid modification. 2023.
38.	Vickers AJ, Cronin AM, Elkin EB, et al. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak, 2008, 8: 53.
39.	Vickers AJ, van Calster B, Steyerberg EW. A simple, step-by-step guide to interpreting decision curve analysis. Diagn Progn Res, 2019, 3: 18.
40.	Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ, 2016, 352: i6.
41.	Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making, 2006, 26(6): 565-574.
42.	Pearl J. An introduction to causal inference. Int J Biostat, 2010, 6(2): Article 7.
43.	Debray TPA, Collins GS, Riley RD, et al. Transparent reporting of multivariable prediction models developed or validated using clustered data (TRIPOD-Cluster): explanation and elaboration. BMJ, 2023, 380: e071058.
44.	李秋萍, 谷鸿秋, 王俊峰. 临床预测模型: TRIPOD报告规范解读—以心血管疾病预测模型QRISK3为例(上). 中国循证心血管医学杂志, 2020, 12(7): 778-782,793.
45.	Reeve K, On BI, Havla J, et al. Prognostic models for predicting clinical disease progression, worsening and activity in people with multiple sclerosis. Cochrane Database Syst Rev, 2023, (9): CD013606.
46.	Bonnett LJ, Snell KIE, Collins GS, et al. Guide to presenting clinical prediction models for use in clinical settings. BMJ, 2019, 365: l737.
47.	Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 2024, 385: e078378.
48.	Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J, 2014, 35(29): 1925-1931.
49.	陈静静, 卿婷玉, 周波. 基于R语言ARIMA模型对我国医疗卫生资源预测研究. 社区医学杂志, 2022, 20(2): 108-115.
50.	谷鸿秋. 临床预测模型的困境与机遇. 中国卒中杂志, 2024, 19(5): 481-487.
51.	医疗大数据白皮书. 2023.

1. Dankers F, Traverso A, Wee L, et al. Prediction modeling methodology. Kubben P, Dumontier M, Dekker A. Fundamentals of clinical data science. Cham (CH): Springer Copyright, 2019.
2. 谷鸿秋, 周支瑞, 章仲恒, 等. 临床预测模型: 基本概念、应用场景及研究思路. 中国循证心血管医学杂志, 2018, 10(12): 1454-1456,1462.
3. Lee YH, Bang H, Kim DJ. How to establish clinical prediction models. Endocrinol Metab (Seoul), 2016, 31(1): 38-44.
4. Bouwmeester W, Zuithoff NP, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med, 2012, 9(5): 1-12.
5. Efthimiou O, Seo M, Chalkou K, et al. Developing clinical prediction models: a step-by-step guide. BMJ, 2024, 386: e078276.
6. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med, 2015, 162(1): 55-63.
7. Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med, 2019, 170(1): 51-58.
8. Ramspek CL, Jager KJ, Dekker FW, et al. External validation of prognostic models: what, why, how, when and where. Clin Kidney J, 2020, 14(1): 49-58.
9. Dankowski T, Ziegler A. Calibrating random forests for probability estimation. Stat Med, 2016, 35(22): 3949-3960.
10. Schuurmans M, Saha A, Alves N, et al. End-to-end prognostication in pancreatic cancer by multimodal deep learning: a retrospective, multicenter study. Eur Radiol, 2025.
11. Altman DG, Royston P. The cost of dichotomising continuous variables. BMJ, 2006, 332(7549): 1080.
12. Fedorov V, Mannino F, Zhang R. Consequences of dichotomization. Pharm Stat, 2009, 8(1): 50-61.
13. Leisman DE, Harhay MO, Lederer DJ, et al. Development and reporting of prediction models: guidance for authors from editors of respiratory, sleep, and critical care journals. Crit Care Med, 2020, 48(5): 623-633.
14. Rhon DI, Teyhen DS, Collins GS, et al. Predictive models for musculoskeletal injury risk: why statistical approach makes all the difference. BMJ Open Sport Exerc Med, 2022, 8(4): e001388.
15. Collins GS, Ogundimu EO, Cook JA, et al. Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model. Stat Med, 2016, 35(23): 4124-4135.
16. Sanchez-Pinto LN, Venable LR, Fahrenbach J, et al. Comparison of variable selection methods for clinical predictive modeling. Int J Med Inform, 2018, 116: 10-17.
17. Taipale H, Schneider-Thoma J, Pinzón-Espinosa J, et al. Representation and outcomes of individuals with schizophrenia seen in everyday practice who are ineligible for randomized clinical trials. JAMA Psychiatry, 2022, 79(3): 210-218.
18. Bennett DA, Landry D, Little J, et al. Systematic review of statistical approaches to quantify, or correct for, measurement error in a continuous exposure in nutritional epidemiology. BMC Med Res Methodol, 2017, 17(1): 146.
19. Yan F, Chen X, Quan X, et al. Association between the stress hyperglycemia ratio and 28-day all-cause mortality in critically ill patients with sepsis: a retrospective cohort study and predictive model establishment based on machine learning. Cardiovasc Diabetol, 2024, 23(1): 163.
20. Barkley SE, Mason JL, Kappelman MM. An assessment of entering freshman medical students' knowledge of and attitudes toward AIDS. Res Med Educ, 1988, 27: 221-225.
21. van der Ploeg T, Austin PC, Steyerberg EW. Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Med Res Methodol, 2014, 14: 137.
22. Infante G, Miceli R, Ambrogi F. Sample size and predictive performance of machine learning methods with survival data: a simulation study. Stat Med, 2023, 42(30): 5657-5675.
23. Sterne JA, White IR, Carlin JB, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ, 2009, 338: b2393.
24. Sisk R, Sperrin M, Peek N, et al. Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: a simulation study. Stat Methods Med Res, 2023, 32(8): 1461-1477.
25. Schafer JL. Multiple imputation: a primer. Stat Methods Med Res, 1999, 8(1): 3-15.
26. Deo RC. Machine learning in medicine. Circulation, 2015, 132(20): 1920-1930.
27. Lo Vercio L, Amador K, Bannister JJ, et al. Supervised machine learning tools: a tutorial for clinicians. J Neural Eng, 2020, 17(6): abbff2.
28. Andaur Navarro CL, Damen JAA, Takada T, et al. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ, 2021, 375: n2281.
29. Putter H, Fiocco M, Geskus RB. Tutorial in biostatistics: competing risks and multi-state models. Stat Med, 2007, 26(11): 2389-2430.
30. Steyerberg EW, Eijkemans MJ, Habbema JD. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol, 1999, 52(10): 935-942.
31. Van Calster B, van Smeden M, De Cock B, et al. Regression shrinkage methods for clinical prediction models do not guarantee improved performance: simulation study. Stat Methods Med Res, 2020, 29(11): 3166-3178.
32. Riley RD, Snell KIE, Martin GP, et al. Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small. J Clin Epidemiol, 2021, 132: 88-96.
33. Wood AM, Royston P, White IR. The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data. Biom J, 2015, 57(4): 614-632.
34. Steyerberg EW. Validation in prediction research: the waste by data splitting. J Clin Epidemiol, 2018, 103: 131-133.
35. Collins GS, Dhiman P, Ma J, et al. Evaluation of clinical prediction models (part 1): from development to external validation. BMJ, 2024, 384: e074819.
36. Lynam AL, Dennis JM, Owen KR, et al. Logistic regression has similar performance to optimised machine learning algorithms in a clinical setting: application to the discrimination between type 1 and type 2 diabetes in young adults. Diagn Progn Res, 2020, 4: 6.
37. National Institute for Health and Care Excellence: Guidelines. Cardiovascular disease: risk assessment and reduction, including lipid modification. 2023.
38. Vickers AJ, Cronin AM, Elkin EB, et al. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak, 2008, 8: 53.
39. Vickers AJ, van Calster B, Steyerberg EW. A simple, step-by-step guide to interpreting decision curve analysis. Diagn Progn Res, 2019, 3: 18.
40. Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ, 2016, 352: i6.
41. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making, 2006, 26(6): 565-574.
42. Pearl J. An introduction to causal inference. Int J Biostat, 2010, 6(2): Article 7.
43. Debray TPA, Collins GS, Riley RD, et al. Transparent reporting of multivariable prediction models developed or validated using clustered data (TRIPOD-Cluster): explanation and elaboration. BMJ, 2023, 380: e071058.
44. 李秋萍, 谷鸿秋, 王俊峰. 临床预测模型: TRIPOD报告规范解读—以心血管疾病预测模型QRISK3为例(上). 中国循证心血管医学杂志, 2020, 12(7): 778-782,793.
45. Reeve K, On BI, Havla J, et al. Prognostic models for predicting clinical disease progression, worsening and activity in people with multiple sclerosis. Cochrane Database Syst Rev, 2023, (9): CD013606.
46. Bonnett LJ, Snell KIE, Collins GS, et al. Guide to presenting clinical prediction models for use in clinical settings. BMJ, 2019, 365: l737.
47. Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 2024, 385: e078378.
48. Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J, 2014, 35(29): 1925-1931.
49. 陈静静, 卿婷玉, 周波. 基于R语言ARIMA模型对我国医疗卫生资源预测研究. 社区医学杂志, 2022, 20(2): 108-115.
50. 谷鸿秋. 临床预测模型的困境与机遇. 中国卒中杂志, 2024, 19(5): 481-487.
51. 医疗大数据白皮书. 2023.

Chinese Journal of Evidence-Based Medicine

Latest ArticlesInterpretation of the step-by-step guide for developing clinical prediction models in 2024

Abstract Full text Figures/Tables Video References Cited by

Format

Content