Artificial intelligence-based diagnostic test accuracy studies: methodological quality assessment and reporting guidelines_Chinese Journal of Evidence-Based Medicine

Authors：

GAO Ge ¹ , CUI Xinxin ¹ , ZENG Mengyu ² , ZENG Wei ¹ , GUO Jixiang ² , ZHANG Tao ³ ,  TANG Wei ¹ ,  LIU Chang ¹

1. State Key Laboratory of Oral Diseases & National Center for Stomatology & National Clinical Research Center for Oral Diseases & "Medicine+Manufacturing" Center, West China Hospital of Stomatology, Sichuan University, Chengdu 610041, P. R. China;
2. Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu 610065, P. R. China;
3. Department of Epidemiology and Health Statistics, West China School of Public Health, Sichuan University, Chengdu 610041, P. R. China;

Corresponding author：

TANG Wei, Email: mydrtw@vip.sina.com; LIU Chang, Email: liu_chang_92@sina.com

Keywords：

Artificial intelligence; Machine learning; Diagnostic test; Risk of bias; Methodological quality assessment; Reporting guidelines

DOI：

10.7507/1672-2531.202310006

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

This paper summarizes the methodological quality assessment tools of artificial intelligence-based diagnostic test accuracy studies, and introduces QUADAS-AI and modified QUADAS-2. Moreover, this paper summarizes reporting guidelines of these studies as well, and then introduces specific reporting standards in AI-centred research, and checklist for AI in dental research.

Citation： GAO Ge, CUI Xinxin, ZENG Mengyu, ZENG Wei, GUO Jixiang, ZHANG Tao, TANG Wei, LIU Chang. Artificial intelligence-based diagnostic test accuracy studies: methodological quality assessment and reporting guidelines. Chinese Journal of Evidence-Based Medicine, 2024, 24(5): 598-604. doi: 10.7507/1672-2531.202310006 Copy

1.	张雨楠, 朱涛, 曾维, 等. 基于机器学习的诊断性试验准确性研究(一): 研究设计. 中国循证医学杂志, 2023, 23(6): 725-730.
2.	熊宇韬, 钟程澜, 曾维, 等. 基于机器学习的诊断试验准确性研究(二): 测量指标. 中国循证医学杂志, 2023, 23(8): 963-969.
3.	Whiting P, Rutjes AW, Reitsma JB, et al. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol, 2003, 3: 25.
4.	刁骧, 艾昌林, 秦莉, 等. QUADAS的制定: 用于系统评价中评价诊断性研究质量的工具. 中国循证医学杂志, 2007, 7(4): 296-306.
5.	Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med, 2011, 155(8): 529-536.
6.	邬兰, 张永, 曾宪涛. QUADAS-2在诊断准确性研究的质量评价工具中的应用. 湖北医药学院学报, 2013, 32(3): 201-208.
7.	曲艳吉, 杨智荣, 孙凤, 等. 偏倚风险评估系列: (六)诊断试验. 中华流行病学杂志, 2018, 39(4): 524-531.
8.	Yang B, Mallett S, Takwoingi Y, et al. QUADAS-C: a tool for assessing risk of bias in comparative diagnostic accuracy studies. Ann Intern Med, 2021, 174(11): 1592-1599.
9.	黄玉香, 沈建通, 刘雨今. 诊断试验准确性比较研究偏倚风险评价工具QUADAS-C解读. 中国循证医学杂志, 2022, 22(9): 1108-1116.
10.	杨秋玉, 陆瑶, 谢欣玲, 等. 诊断试验准确性比较研究的偏倚评估工具—QUADAS-C. 中华流行病学杂志, 2022, 43(6): 938-944.
11.	Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med, 2019, 170(1): 51-58.
12.	陈茹, 王胜锋, 周家琛, 等. 预测模型研究的偏倚风险和适用性评估工具解读. 中华流行病学杂志, 2020, 41(5): 776-781.
13.	Hayden JA, van der Windt DA, Cartwright JL, et al. Assessing bias in studies of prognostic factors. Ann Intern Med, 2013, 158(4): 280-286.
14.	唐少文, 张渊, 陶必林, 等. 偏倚风险评估系列: (七)预后因素研究. 中华流行病学杂志, 2018, 39(7): 1003-1008.
15.	Lee J, Mulder F, Leeflang M, et al. QUAPAS: an adaptation of the QUADAS-2 tool to assess prognostic accuracy studies. Ann Intern Med, 2022, 175(7): 1010-1018.
16.	张妹, 卫建华, 沈建通, 等. 预后准确性研究偏倚风险评价工具QUAPAS解读. 中国循证医学杂志, 2023, 23(4): 457-464.
17.	Sounderajah V, Ashrafian H, Rose S, et al. A quality assessment tool for artificial intelligence-centered diagnostic test accuracy studies: QUADAS-AI. Nat Med, 2021, 27(10): 1663-1665.
18.	Mohammad-Rahimi H, Motamedian SR, Rohban MH, et al. Deep learning for caries detection: a systematic review. J Dent, 2022, 122: 104115.
19.	Collins GS, Dhiman P, Andaur Navarro CL, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open, 2021, 11(7): e048008.
20.	Bossuyt PM, Reitsma JB, Bruns DE, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. BMJ, 2003, 326(7379): 41-44.
21.	刁骧, 秦莉, 姚巡. 迈向完整、准确的诊断准确性研究报告: STARD计划. 中国循证医学杂志, 2006, 6(7): 523-528.
22.	Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ, 2015, 351: h5527.
23.	朱一丹, 李会娟, 武阳丰. 诊断准确性研究报告规范(STARD)2015介绍与解读. 中国循证医学杂志, 2016, 16(6): 730-735.
24.	陈新林, 胡月, 莫传伟, 等. 诊断准确性研究报告指南—STARD2015简介. 中国循证医学杂志, 2016, 16(10): 1227-1230.
25.	Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ, 2015, 350: g7594.
26.	曹煜隆, 单娇, 龚志忠, 等. 个体预后与诊断预测模型研究报告规范—TRIPOD声明解读. 中国循证医学杂志, 2020, 20(4): 492-496.
27.	Nagendrababu V, Abbott P, Duncan HF, et al. Preferred Reporting Items for Diagnostic Accuracy Studies in Endodontics (PRIDASE) guidelines: a development protocol. Int Endod J, 2021, 54(7): 1051-1055.
28.	Collins GS, Moons KGM. Reporting of artificial intelligence prediction models. Lancet, 2019, 393(10181): 1577-1579.
29.	Sounderajah V, Ashrafian H, Aggarwal R, et al. Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: the STARD-AI Steering Group. Nat Med, 2020, 26(6): 807-808.
30.	Sounderajah V, Ashrafian H, Golub RM, et al. Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol. BMJ Open, 2021, 11(6): e047709.
31.	Sounderajah V, Ashrafian H, Karthikesalingam A, et al. Developing specific reporting standards in artificial intelligence centred research. Ann Surg, 2022, 275(3): e547-e548.
32.	Liu X, Rivera SC, Moher D, et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI Extension. BMJ, 2020, 370: m3164.
33.	高亚, 刘明, 李菲, 等. 人工智能干预性临床试验报告规范: CONSORT-AI解读. 中国医药导刊, 2020, 22(10): 687-691.
34.	李子孝, 熊云云, 丁玲玲, 等. 人工智能干预性临床试验报告指南: CONSORT-AI扩展. 中国卒中杂志, 2020, 15(12): 1327-1336.
35.	Rivera SC, Liu X, Chan AW, et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. BMJ, 2020, 370: m3210.
36.	刘明, 高亚, 史纪元, 等. 人工智能干预临床试验研究方案报告规范指南: SPIRIT-AI扩展版(2020)解读. 中国医药导刊, 2020, 22(10): 692-697.
37.	熊云云, 李子孝, 丁玲玲, 等. 基于人工智能干预措施的临床试验方案指南: SPIRIT-AI扩展. 中国卒中杂志, 2020, 15(11): 1228-1238.
38.	Mongan J, Moy L, Kahn CE. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell, 2020, 2(2): e200029.
39.	诸宇佳, 韩慧, 卫建华, 等. 人工智能医学影像研究报告规范: CLAIM检查清单解读. 中国循证医学杂志, 2023, 23(12): 1478-1484.
40.	Schwendicke F, Singh T, Lee JH, et al. Artificial intelligence in dental research: checklist for authors, reviewers, readers. J Dent, 2021, 107: 103610.
41.	Huang H, Zheng O, Wang D, et al. ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model. Int J Oral Sci, 2023, 15(1): 29.
42.	国际医学期刊编辑委员会, 张俊彦, 于笑天, 等. 学术研究实施与报告和医学期刊编辑与发表的推荐规范(一). 法医学杂志, 2020, 36(5): 741-746.

1. 张雨楠, 朱涛, 曾维, 等. 基于机器学习的诊断性试验准确性研究(一): 研究设计. 中国循证医学杂志, 2023, 23(6): 725-730.
2. 熊宇韬, 钟程澜, 曾维, 等. 基于机器学习的诊断试验准确性研究(二): 测量指标. 中国循证医学杂志, 2023, 23(8): 963-969.
3. Whiting P, Rutjes AW, Reitsma JB, et al. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol, 2003, 3: 25.
4. 刁骧, 艾昌林, 秦莉, 等. QUADAS的制定: 用于系统评价中评价诊断性研究质量的工具. 中国循证医学杂志, 2007, 7(4): 296-306.
5. Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med, 2011, 155(8): 529-536.
6. 邬兰, 张永, 曾宪涛. QUADAS-2在诊断准确性研究的质量评价工具中的应用. 湖北医药学院学报, 2013, 32(3): 201-208.
7. 曲艳吉, 杨智荣, 孙凤, 等. 偏倚风险评估系列: (六)诊断试验. 中华流行病学杂志, 2018, 39(4): 524-531.
8. Yang B, Mallett S, Takwoingi Y, et al. QUADAS-C: a tool for assessing risk of bias in comparative diagnostic accuracy studies. Ann Intern Med, 2021, 174(11): 1592-1599.
9. 黄玉香, 沈建通, 刘雨今. 诊断试验准确性比较研究偏倚风险评价工具QUADAS-C解读. 中国循证医学杂志, 2022, 22(9): 1108-1116.
10. 杨秋玉, 陆瑶, 谢欣玲, 等. 诊断试验准确性比较研究的偏倚评估工具—QUADAS-C. 中华流行病学杂志, 2022, 43(6): 938-944.
11. Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med, 2019, 170(1): 51-58.
12. 陈茹, 王胜锋, 周家琛, 等. 预测模型研究的偏倚风险和适用性评估工具解读. 中华流行病学杂志, 2020, 41(5): 776-781.
13. Hayden JA, van der Windt DA, Cartwright JL, et al. Assessing bias in studies of prognostic factors. Ann Intern Med, 2013, 158(4): 280-286.
14. 唐少文, 张渊, 陶必林, 等. 偏倚风险评估系列: (七)预后因素研究. 中华流行病学杂志, 2018, 39(7): 1003-1008.
15. Lee J, Mulder F, Leeflang M, et al. QUAPAS: an adaptation of the QUADAS-2 tool to assess prognostic accuracy studies. Ann Intern Med, 2022, 175(7): 1010-1018.
16. 张妹, 卫建华, 沈建通, 等. 预后准确性研究偏倚风险评价工具QUAPAS解读. 中国循证医学杂志, 2023, 23(4): 457-464.
17. Sounderajah V, Ashrafian H, Rose S, et al. A quality assessment tool for artificial intelligence-centered diagnostic test accuracy studies: QUADAS-AI. Nat Med, 2021, 27(10): 1663-1665.
18. Mohammad-Rahimi H, Motamedian SR, Rohban MH, et al. Deep learning for caries detection: a systematic review. J Dent, 2022, 122: 104115.
19. Collins GS, Dhiman P, Andaur Navarro CL, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open, 2021, 11(7): e048008.
20. Bossuyt PM, Reitsma JB, Bruns DE, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. BMJ, 2003, 326(7379): 41-44.
21. 刁骧, 秦莉, 姚巡. 迈向完整、准确的诊断准确性研究报告: STARD计划. 中国循证医学杂志, 2006, 6(7): 523-528.
22. Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ, 2015, 351: h5527.
23. 朱一丹, 李会娟, 武阳丰. 诊断准确性研究报告规范(STARD)2015介绍与解读. 中国循证医学杂志, 2016, 16(6): 730-735.
24. 陈新林, 胡月, 莫传伟, 等. 诊断准确性研究报告指南—STARD2015简介. 中国循证医学杂志, 2016, 16(10): 1227-1230.
25. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ, 2015, 350: g7594.
26. 曹煜隆, 单娇, 龚志忠, 等. 个体预后与诊断预测模型研究报告规范—TRIPOD声明解读. 中国循证医学杂志, 2020, 20(4): 492-496.
27. Nagendrababu V, Abbott P, Duncan HF, et al. Preferred Reporting Items for Diagnostic Accuracy Studies in Endodontics (PRIDASE) guidelines: a development protocol. Int Endod J, 2021, 54(7): 1051-1055.
28. Collins GS, Moons KGM. Reporting of artificial intelligence prediction models. Lancet, 2019, 393(10181): 1577-1579.
29. Sounderajah V, Ashrafian H, Aggarwal R, et al. Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: the STARD-AI Steering Group. Nat Med, 2020, 26(6): 807-808.
30. Sounderajah V, Ashrafian H, Golub RM, et al. Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol. BMJ Open, 2021, 11(6): e047709.
31. Sounderajah V, Ashrafian H, Karthikesalingam A, et al. Developing specific reporting standards in artificial intelligence centred research. Ann Surg, 2022, 275(3): e547-e548.
32. Liu X, Rivera SC, Moher D, et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI Extension. BMJ, 2020, 370: m3164.
33. 高亚, 刘明, 李菲, 等. 人工智能干预性临床试验报告规范: CONSORT-AI解读. 中国医药导刊, 2020, 22(10): 687-691.
34. 李子孝, 熊云云, 丁玲玲, 等. 人工智能干预性临床试验报告指南: CONSORT-AI扩展. 中国卒中杂志, 2020, 15(12): 1327-1336.
35. Rivera SC, Liu X, Chan AW, et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. BMJ, 2020, 370: m3210.
36. 刘明, 高亚, 史纪元, 等. 人工智能干预临床试验研究方案报告规范指南: SPIRIT-AI扩展版(2020)解读. 中国医药导刊, 2020, 22(10): 692-697.
37. 熊云云, 李子孝, 丁玲玲, 等. 基于人工智能干预措施的临床试验方案指南: SPIRIT-AI扩展. 中国卒中杂志, 2020, 15(11): 1228-1238.
38. Mongan J, Moy L, Kahn CE. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell, 2020, 2(2): e200029.
39. 诸宇佳, 韩慧, 卫建华, 等. 人工智能医学影像研究报告规范: CLAIM检查清单解读. 中国循证医学杂志, 2023, 23(12): 1478-1484.
40. Schwendicke F, Singh T, Lee JH, et al. Artificial intelligence in dental research: checklist for authors, reviewers, readers. J Dent, 2021, 107: 103610.
41. Huang H, Zheng O, Wang D, et al. ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model. Int J Oral Sci, 2023, 15(1): 29.
42. 国际医学期刊编辑委员会, 张俊彦, 于笑天, 等. 学术研究实施与报告和医学期刊编辑与发表的推荐规范(一). 法医学杂志, 2020, 36(5): 741-746.

Previous Article
Evidence map of effectiveness of dietary sugar reduction
Next Article
Large scale simple clinical trial designs supported by real world data

Chinese Journal of Evidence-Based Medicine

Artificial intelligence-based diagnostic test accuracy studies: methodological quality assessment and reporting guidelines

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content