AI-based diagnostic accuracy and prognosis research reporting guideline: interpretation of the TRIPOD+AI Statement_CHINESE JOURNAL OF BASES AND CLINICS IN GENERAL SURGERY

Authors：

SU Wen , LAI Zepeng , JIANG Haolin , ZENG Ziqian ,  CHEN Weizhong

Department of Epidemiology and Statistics, School of Public Health, Chengdu Medical College, Chengdu 610500, P. R. China;

Corresponding author：

CHEN Weizhong, Email: wejone@126.com

Keywords：

TRIPOD+AI; machine learning; machine learning model; reporting guideline; predictive model

DOI：

10.7507/1007-9424.202411127

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

With the increasing availability of clinical and biomedical big data, machine learning is being widely used in scientific research and academic papers. It integrates various types of information to predict individual health outcomes. However, deficiencies in reporting key information have gradually emerged. These include issues like data bias, model fairness across different groups, and problems with data quality and applicability. Maintaining predictive accuracy and interpretability in real-world clinical settings is also a challenge. This increases the complexity of safely and effectively applying predictive models to clinical practice. To address these problems, TRIPOD+AI (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis+artificial intelligence) introduces a reporting standard for machine learning models. It is based on TRIPOD and aims to improve transparency, reproducibility, and health equity. These improvements enhance the quality of machine learning model applications. Currently, research on prediction models based on machine learning is rapidly increasing. To help domestic readers better understand and apply TRIPOD+AI, we provide examples and interpretations. We hope this will support researchers in improving the quality of their reports.

1.	Au EH, Francis A, Bernier-Jean A, et al. Prediction modeling-part 1: regression modeling. Kidney Int, 2020, 97(5): 877-884.
2.	Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 2017, 542(7639): 115-118.
3.	Deo RC. Machine learning in medicine. Circulation, 2015, 132(20): 1920-1930.
4.	Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ, 2015, 350: g7594. doi: 10.1136/bmj.g7594.
5.	Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 2024, 385: e078378. doi: 10.1136/bmj-2023-078378.
6.	De Filippo O, Cammann VL, Pancotti C, et al. Machine learning-based prediction of in-hospital death for patients with takotsubo syndrome: the InterTAK-ML model. Eur J Heart Fail, 2023, 25(12): 2299-2311.
7.	曹煜隆, 单娇, 龚志忠, 等. 个体预后与诊断预测模型研究报告规范—TRIPOD声明解读. 中国循证医学杂志, 2020, 20(4): 492-496.
8.	Van Calster B, McLernon DJ, van Smeden M, et al. Calibration: the Achilles heel of predictive analytics. BMC Med, 2019, 17(1): 230. doi: 10.1186/s12916-019-1466-7.
9.	Jamarani A, Haddadi S, Sarvizadeh R, et al. Big data and predictive analytics: a systematic review of applications. Artificial Intelligence Review, 2024, 57(7):.
10.	Adler-Milstein J, Chen JH, Dhaliwal G. Next-generation artificial intelligence for diagnosis: from predicting diagnostic labels to “wayfinding”. JAMA, 2021, 326(24): 2467-2468.
11.	Smith BT, Smith PM, Harper S, et al. Reducing social inequalities in health: the role of simulation modelling in chronic disease epidemiology to evaluate the impact of population health interventions. J Epidemiol Community Health, 2014, 68(4): 384-389.
12.	Yang C, Kors JA, Ioannou S, et al. Trends in the conduct and reporting of clinical prediction model development and validation: a systematic review. J Am Med Inform Assoc, 2022, 29(5): 983-989.
13.	Bhandari N, Walambe R, Kotecha K, et al. A comprehensive survey on computational learning methods for analysis of gene expression data. Front Mol Biosci, 2022, 9: 907150. doi: 10.3389/fmolb.2022.907150.
14.	Chen P, Wu L, Lei Wang. AI fairness in data management and analytics: a review on challenges, methodologies and applications. Appl Sci, 2023, 13(18): 10258. doi: 10.3390/app131810258.
15.	Liu F, Panagiotakos D. Real-world data: a brief review of the methods, applications, challenges and opportunities. BMC Med Res Methodol, 2022, 22(1): 287. doi: 10.1186/s12874-022-01768-6.
16.	Ferrara E. Fairness and bias in artificial intelligence: a brief survey of sources, impacts, and mitigation strategies. Sci, 2024, 6(1): 3. doi: 10.3390/sci6010003.
17.	Christodoulou E, van Smeden M, Edlinger M, et al. Adaptive sample size determination for the development of clinical prediction models. Diagn Progn Res, 2021, 5(1): 6. doi: 10.1186/s41512-021-00096-5.
18.	Ng W, Minasny B, Mendes WDS, et al. The influence of training sample size on the accuracy of deep learning models for the prediction of soil properties with near-infrared spectroscopy data. SOIL, 6: 565-578. https://doi.org/10.5194/soil-6-565-2020, 2020.
19.	Debray TPA, Collins GS, Riley RD, et al. Transparent reporting of multivariable prediction models developed or validated using clustered data (TRIPOD-Cluster): explanation and elaboration. BMJ, 2023, 380: e071058. doi: 10.1136/bmj-2022-071058.
20.	陶立元, 刘珏. 基于多源数据的个体预后或诊断多因素预测模型报告规范(TRIPOD-Cluster)解读. 中华医学杂志, 2023, 103(36): 2893-2897.
21.	Gerds TA, Kattan MW. Medical risk prediction. New York: Chapman and Hall/CRC, 2021: 312.
22.	Cusworth S, Gkoutos GV, Acharjee A. A novel generative adversarial networks modelling for the class imbalance problem in high dimensional omics data. BMC Med Inform Decis Mak, 2024, 24(1): 90. doi: 10.1186/s12911-024-02487-2.
23.	Rajaraman S, Ganesan P, Antani S. Deep learning model calibration for improving performance in class-imbalanced medical image classification tasks. PLoS One, 2022, 17(1): e0262838. doi: 10.1371/journal.pone.0262838.
24.	Liu S, Vicente LN. Accuracy and fairness trade-offs in machine learning: a stochastic multi-objective approach. Comput Manag Sci, 2022, 19: 513-537.
25.	Zou KH, O’Malley AJ, Mauri L. Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation, 2007, 115(5): 654-657.
26.	Obermeyer Z, Emanuel EJ. Predicting the future - big data, machine learning, and clinical medicine. N Engl J Med, 2016, 375(13): 1216-1219.
27.	Finlayson SG, Subbaswamy A, Singh K, et al. The clinician and dataset shift in artificial intelligence. N Engl J Med, 2021, 385(3): 283-286.
28.	Probst P, Wright M, Boulesteix AL. Hyperparameters and tuning strategies for random forest. WIREs Data Mining Knowl Discov, 2018, 9.
29.	Talic S, Shah S, Wild H, et al. Effectiveness of public health measures in reducing the incidence of covid-19, SARS-CoV-2 transmission, and covid-19 mortality: systematic review and meta-analysis. BMJ, 2021, 375: e068302.
30.	Chen IY, Joshi S, Ghassemi M. Treating health disparities with artificial intelligence. Nat Med, 2020, 26(1): 16-17.
31.	Mazzolenis ME, Bulat E, Schatman ME, et al. The ethical stewardship of artificial intelligence in chronic pain and headache: a narrative review. Curr Pain Headache Rep, 2024, 28(8): 785-792.
32.	Gil-Fuster E, Eisert J, Bravo-Prieto C. Understanding quantum machine learning also requires rethinking generalization. Nat Commun, 2024, 15(1): 2277. doi: 10.1038/s41467-024-45882-z.
33.	de Jong J, Emon MA, Wu P, et al. Deep learning for clustering of multivariate clinical patient trajectories with missing values. Gigascience, 2019, 8(11): giz134. doi: 10.1093/gigascience/giz134.

1. Au EH, Francis A, Bernier-Jean A, et al. Prediction modeling-part 1: regression modeling. Kidney Int, 2020, 97(5): 877-884.
2. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 2017, 542(7639): 115-118.
3. Deo RC. Machine learning in medicine. Circulation, 2015, 132(20): 1920-1930.
4. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ, 2015, 350: g7594. doi: 10.1136/bmj.g7594.
5. Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 2024, 385: e078378. doi: 10.1136/bmj-2023-078378.
6. De Filippo O, Cammann VL, Pancotti C, et al. Machine learning-based prediction of in-hospital death for patients with takotsubo syndrome: the InterTAK-ML model. Eur J Heart Fail, 2023, 25(12): 2299-2311.
7. 曹煜隆, 单娇, 龚志忠, 等. 个体预后与诊断预测模型研究报告规范—TRIPOD声明解读. 中国循证医学杂志, 2020, 20(4): 492-496.
8. Van Calster B, McLernon DJ, van Smeden M, et al. Calibration: the Achilles heel of predictive analytics. BMC Med, 2019, 17(1): 230. doi: 10.1186/s12916-019-1466-7.
9. Jamarani A, Haddadi S, Sarvizadeh R, et al. Big data and predictive analytics: a systematic review of applications. Artificial Intelligence Review, 2024, 57(7):.
10. Adler-Milstein J, Chen JH, Dhaliwal G. Next-generation artificial intelligence for diagnosis: from predicting diagnostic labels to “wayfinding”. JAMA, 2021, 326(24): 2467-2468.
11. Smith BT, Smith PM, Harper S, et al. Reducing social inequalities in health: the role of simulation modelling in chronic disease epidemiology to evaluate the impact of population health interventions. J Epidemiol Community Health, 2014, 68(4): 384-389.
12. Yang C, Kors JA, Ioannou S, et al. Trends in the conduct and reporting of clinical prediction model development and validation: a systematic review. J Am Med Inform Assoc, 2022, 29(5): 983-989.
13. Bhandari N, Walambe R, Kotecha K, et al. A comprehensive survey on computational learning methods for analysis of gene expression data. Front Mol Biosci, 2022, 9: 907150. doi: 10.3389/fmolb.2022.907150.
14. Chen P, Wu L, Lei Wang. AI fairness in data management and analytics: a review on challenges, methodologies and applications. Appl Sci, 2023, 13(18): 10258. doi: 10.3390/app131810258.
15. Liu F, Panagiotakos D. Real-world data: a brief review of the methods, applications, challenges and opportunities. BMC Med Res Methodol, 2022, 22(1): 287. doi: 10.1186/s12874-022-01768-6.
16. Ferrara E. Fairness and bias in artificial intelligence: a brief survey of sources, impacts, and mitigation strategies. Sci, 2024, 6(1): 3. doi: 10.3390/sci6010003.
17. Christodoulou E, van Smeden M, Edlinger M, et al. Adaptive sample size determination for the development of clinical prediction models. Diagn Progn Res, 2021, 5(1): 6. doi: 10.1186/s41512-021-00096-5.
18. Ng W, Minasny B, Mendes WDS, et al. The influence of training sample size on the accuracy of deep learning models for the prediction of soil properties with near-infrared spectroscopy data. SOIL, 6: 565-578. https://doi.org/10.5194/soil-6-565-2020, 2020.
19. Debray TPA, Collins GS, Riley RD, et al. Transparent reporting of multivariable prediction models developed or validated using clustered data (TRIPOD-Cluster): explanation and elaboration. BMJ, 2023, 380: e071058. doi: 10.1136/bmj-2022-071058.
20. 陶立元, 刘珏. 基于多源数据的个体预后或诊断多因素预测模型报告规范(TRIPOD-Cluster)解读. 中华医学杂志, 2023, 103(36): 2893-2897.
21. Gerds TA, Kattan MW. Medical risk prediction. New York: Chapman and Hall/CRC, 2021: 312.
22. Cusworth S, Gkoutos GV, Acharjee A. A novel generative adversarial networks modelling for the class imbalance problem in high dimensional omics data. BMC Med Inform Decis Mak, 2024, 24(1): 90. doi: 10.1186/s12911-024-02487-2.
23. Rajaraman S, Ganesan P, Antani S. Deep learning model calibration for improving performance in class-imbalanced medical image classification tasks. PLoS One, 2022, 17(1): e0262838. doi: 10.1371/journal.pone.0262838.
24. Liu S, Vicente LN. Accuracy and fairness trade-offs in machine learning: a stochastic multi-objective approach. Comput Manag Sci, 2022, 19: 513-537.
25. Zou KH, O’Malley AJ, Mauri L. Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation, 2007, 115(5): 654-657.
26. Obermeyer Z, Emanuel EJ. Predicting the future - big data, machine learning, and clinical medicine. N Engl J Med, 2016, 375(13): 1216-1219.
27. Finlayson SG, Subbaswamy A, Singh K, et al. The clinician and dataset shift in artificial intelligence. N Engl J Med, 2021, 385(3): 283-286.
28. Probst P, Wright M, Boulesteix AL. Hyperparameters and tuning strategies for random forest. WIREs Data Mining Knowl Discov, 2018, 9.
29. Talic S, Shah S, Wild H, et al. Effectiveness of public health measures in reducing the incidence of covid-19, SARS-CoV-2 transmission, and covid-19 mortality: systematic review and meta-analysis. BMJ, 2021, 375: e068302.
30. Chen IY, Joshi S, Ghassemi M. Treating health disparities with artificial intelligence. Nat Med, 2020, 26(1): 16-17.
31. Mazzolenis ME, Bulat E, Schatman ME, et al. The ethical stewardship of artificial intelligence in chronic pain and headache: a narrative review. Curr Pain Headache Rep, 2024, 28(8): 785-792.
32. Gil-Fuster E, Eisert J, Bravo-Prieto C. Understanding quantum machine learning also requires rethinking generalization. Nat Commun, 2024, 15(1): 2277. doi: 10.1038/s41467-024-45882-z.
33. de Jong J, Emon MA, Wu P, et al. Deep learning for clustering of multivariate clinical patient trajectories with missing values. Gigascience, 2019, 8(11): giz134. doi: 10.1093/gigascience/giz134.

CHINESE JOURNAL OF BASES AND CLINICS IN GENERAL SURGERY

Latest ArticlesAI-based diagnostic accuracy and prognosis research reporting guideline: interpretation of the TRIPOD+AI Statement

Abstract Full text Figures/Tables Video References Cited by

Format

Content