The application of large language models in the field of evidence-based medicine_Chinese Journal of Evidence-Based Medicine

Authors：

LUO Xufei ¹ , LYU Han ² , SHI Qianling ³ , WANG Zijun ¹ , LIU Hui ¹ , ZHU Di ⁴ , WANG Ye ⁴ ,  CHEN Yaolong ^1,3,4,5,6,7

1. School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000, P. R. China;
2. Beijing Friendship Hospital, Capital Medical University, Beijing 100050, P. R. China;
3. The First Clinical Medical College, Lanzhou University, Lanzhou 730000, P. R. China;
4. School of Public Health, Lanzhou University, Lanzhou 730000, P. R. China;
5. Institute of Health Data Science, Lanzhou University, Lanzhou 730000, P. R. China;
6. WHO Collaborating Centre for Guideline Implementation and Knowledge Translation, Lanzhou University, Lanzhou 730000, P. R. China;
7. Research Unit of Evidence-based Evaluation and Guidelines, Chinese Academy of Medical Sciences, School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000, P. R. China;

Corresponding author：

CHEN Yaolong, Email: chevidence@lzu.edu.cn

Keywords：

Large language model; Evidence-based medicine; Systematic review; Clinical practice guideline; ChatGPT; Quality evaluation

DOI：

10.7507/1672-2531.202312067

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Large Language Models (LLMs) are highly sophisticated deep learning models pre-trained on massive datasets, with ChatGPT representing a prominent application of LLMs in the field of generative models. Since the release of ChatGPT at the end of 2022, generative chatbots have become widely employed across various medical disciplines. As a crucial discipline guiding clinical practices, the usage of generative chatbots like ChatGPT in Evidence-Based Medicine (EBM) is gradually increasing. However, the potential, challenges, and intricacies of their application in the domain of EBM remain unclear. This paper aims to explore and discuss the prospects, challenges, and considerations associated with the application of ChatGPT in the field of EBM through a review of relevant literature. The discussion spans four aspects: evidence generation, synthesis, assessment, dissemination, and implementation, providing researchers with insights into the latest developments and future research suggestions.

Citation： LUO Xufei, LYU Han, SHI Qianling, WANG Zijun, LIU Hui, ZHU Di, WANG Ye, CHEN Yaolong. The application of large language models in the field of evidence-based medicine. Chinese Journal of Evidence-Based Medicine, 2024, 24(4): 373-377. doi: 10.7507/1672-2531.202312067 Copy

1.	Pratim RP. ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Inter Things Cyber-Phy Sys, 2023, 3: 121-154.
2.	Loh E. ChatGPT and generative AI chatbots: challenges and opportunities for science, medicine and medical leaders. BMJ Lead, 2023, 5: leader-2023-000797.2023r-2797r.
3.	Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell, 2023, 6: 1169595.
4.	马武仁, 弓孟春, 戴辉, 等. 以ChatGPT为代表的大语言模型在临床医学中的应用综述. 医学信息学杂志, 2023, 44(7): 9-17.
5.	徐璐璐, 洪贇, 叶鹰. ChatGPT及GPT类技术的医学信息学应用前景探讨. 情报理论与实践, 2023, 46(6): 38-42.
6.	Fleming TR, DeMets DL. Surrogate end points in clinical trials: are we being misled. Ann Intern Med, 1996, 125(7): 605-613.
7.	Wei Y, Guo X. Revolutionizing clinical experimental protocol design through the ChatGPT technology. Am J Med Sci, 2023, 366(6): 468-470.
8.	Peikos G, Symeonidis S, Kasela P, et al. Utilizing ChatGPT to enhance clinical trial enrollment. arXiv preprint, 2023, 6: arXiv: 2306.02077.
9.	Methnani J, Latiri I, Dergaa I, et al. ChatGPT for sample-size calculation in sports medicine and exercise sciences: a cautionary note. Int J Sports Physiol Perform, 2023, 18(10): 1219-1223.
10.	Hernandez AF, Lindsell CJ. The future of clinical trials: artificial to augmented to applied intelligence. JAMA, 2023, 330(21): 2061-2063.
11.	Hassani H, Silva ES. The role of ChatGPT in data science: how AI-assisted conversational interfaces are revolutionizing the field. Big Data Cogn Comput, 2023, 7(2): 62.
12.	Sanmarchi F, Bucci A, Nuzzolese AG, et al. A step-by-step researcher's guide to the use of an AI-based transformer in epidemiology: an exploratory analysis of ChatGPT using the STROBE checklist for observational studies. Z Gesundh Wiss, 2023, 5: 1-36.
13.	Muka T, Glisic M, Milic J, et al. A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research. Eur J Epidemiol, 2020, 35(1): 49-60.
14.	Siddaway AP, Wood AM, Hedges LV. How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annu Rev Psychol, 2019, 70: 747-770.
15.	Tawfik GM, Dila KAS, Mohamed MYF, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health, 2019, 47: 46.
16.	Mahuli SA, Rai A, Mahuli AV, et al. Application ChatGPT in conducting systematic reviews and meta-analyses. Br Dent J, 2023, 235(2): 90-92.
17.	Qureshi R, Shaughnessy D, Gill KAR, et al. Are ChatGPT and large language models "the answer" to bringing us closer to systematic review automation. Syst Rev, 2023, 12(1): 72.
18.	van Dijk SHB, Brusse-Keizer MGJ, Bucsán CC, et al. Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open, 2023, 13(7): e072254.
19.	Atkinson CF. Cheap, quick, and rigorous: artificial intelligence and the systematic literature review. Social Sci Comp Review, 2023: 08944393231196281.
20.	陈润生. 医疗大数据结合大语言模型的应用展望. 四川大学学报(医学版), 2023, 54(5): 855-856.
21.	Scott AM, Glasziou P, Clark J. We extended the 2-week systematic review (2 week SR) methodology to larger, more complex systematic reviews: a case series. J Clin Epidemiol, 2023, 5(157): 112-119.
22.	Huang J, Tan M. The role of ChatGPT in scientific communication: writing better scientific review articles. Am J Cancer Res, 2023, 13(4): 1148-1154.
23.	Alshami A, Elsayed M, Ali E, et al. Harnessing the power of ChatGPT for automating systematic review process: methodology, case study, limitations, and future directions. Systems, 2023, 11(7): 351.
24.	Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc, 2016, 23(1): 193-201.
25.	Kim SG. Using ChatGPT for language editing in scientific articles. Maxillofac Plast Reconstr Surg, 2023, 45(1): 13.
26.	Gao CA, Howard FM, Markov NS, et al. Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. NPJ Digit Med, 2023, 6(1): 75.
27.	Kim JK, Chua M, Rickard M, et al. ChatGPT and large language model (LLM) chatbots: The current state of acceptability and a proposal for guidelines on utilization in academic medicine. J Pediatr Urol, 2023, 19(5): 598-604.
28.	Nuzula IF, Amri MM. Will ChatGPT bring a new paradigm to HR world. A critical opinion article. J Manage Stud Devel, 2023, 2(2): 142-161.
29.	Peng Y, Rousseau JF, Shortliffe EH, et al. AI-generated text may have a role in evidence-based medicine. Nat Med, 2023, 29(7): 1593-1594.
30.	Higgins JP, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ, 2011, 343: d5928.
31.	Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ, 2019, 366: l4898.
32.	Schulz KF, Altman DG, Moher D, et al. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ, 2010, 340: c332.
33.	Pitre T, Jassal T, Talukdar JR, et al. ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: a methods study. medRxiv preprint, 2023.
34.	Roberts RH, Ali SR, Hutchings HA, et al. Comparative study of ChatGPT and human evaluators on the assessment of medical literature according to recognised reporting standards. BMJ Health Care Inform, 2023, 30(1): e100830.
35.	Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Clinical practice guidelines we can trust. Washington (DC): National Academies Press, 2011.
36.	Lahat A, Shachar E, Avidan B, et al. Evaluating the use of large language model in identifying top research questions in gastroenterology. Sci Rep, 2023, 13(1): 4164.
37.	Serj Hunt. Use ChatGPT to find contact details of experts in any topic. Youtube, 2024.
38.	The New York Times. Do you have a conflict of interest? This robotic assistant may find it first. The New York Times, 2020.
39.	Sohail SS, Madsen DØ, Himeur Y, et al. Using ChatGPT to navigate ambivalent and contradictory research findings on artificial intelligence. Front Artif Intell, 2023, 6: 1195797.
40.	Hamed E, Eid A, Alberry M. Exploring ChatGPT's potential in facilitating adaptation of clinical guidelines: a case study of diabetic ketoacidosis guidelines. Cureus, 2023, 15(5): e38784.
41.	Haverkamp W, Tennenbaum J, Strodthoff N. ChatGPT fails the test of evidence-based medicine. Eur Heart J Digit Health, 2023, 4(5): 366-367.
42.	Gaggioli A. Ethics: disclose use of AI in scientific manuscripts. Nature, 2023, 614(7948): 413.
43.	Ali K, Barhom N, Tamimi F, et al. ChatGPT-A double-edged sword for healthcare education. Implications for assessments of dental students. Eur J Dent Educ, 2024, 28(1): 206-211.
44.	Palal D, Ghonge S, Jadav V, et al. ChatGPT: a double-edged sword? Health Serv Insights, 2023, 16: 11786329231174338.
45.	Shen Y, Heacock L, Elias J, et al. ChatGPT and other large language models are double-edged swords. Radiology, 2023, 307(2): e230163.
46.	Cacciamani GE, Collins GS, Gill IS. ChatGPT: standard reporting guidelines for responsible use. Nature, 2023, 618(7964): 238.
47.	Huo B, Cacciamani GE, Collins GS, et al. Reporting standards for the use of large language model-linked chatbots for health advice. Nat Med, 2023, 29(12): 2988.
48.	Luo X, Estill J, Chen Y. The use of ChatGPT in medical research: do we need a reporting guideline. Int J Surg, 2023, 109(12): 3750-3751.
49.	李佩芳, 陈佳丽, 宁宁, 等. ChatGPT在医学领域的应用进展及思考. 华西医学, 2023, 38(10): 1456-1460.

1. Pratim RP. ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Inter Things Cyber-Phy Sys, 2023, 3: 121-154.
2. Loh E. ChatGPT and generative AI chatbots: challenges and opportunities for science, medicine and medical leaders. BMJ Lead, 2023, 5: leader-2023-000797.2023r-2797r.
3. Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell, 2023, 6: 1169595.
4. 马武仁, 弓孟春, 戴辉, 等. 以ChatGPT为代表的大语言模型在临床医学中的应用综述. 医学信息学杂志, 2023, 44(7): 9-17.
5. 徐璐璐, 洪贇, 叶鹰. ChatGPT及GPT类技术的医学信息学应用前景探讨. 情报理论与实践, 2023, 46(6): 38-42.
6. Fleming TR, DeMets DL. Surrogate end points in clinical trials: are we being misled. Ann Intern Med, 1996, 125(7): 605-613.
7. Wei Y, Guo X. Revolutionizing clinical experimental protocol design through the ChatGPT technology. Am J Med Sci, 2023, 366(6): 468-470.
8. Peikos G, Symeonidis S, Kasela P, et al. Utilizing ChatGPT to enhance clinical trial enrollment. arXiv preprint, 2023, 6: arXiv: 2306.02077.
9. Methnani J, Latiri I, Dergaa I, et al. ChatGPT for sample-size calculation in sports medicine and exercise sciences: a cautionary note. Int J Sports Physiol Perform, 2023, 18(10): 1219-1223.
10. Hernandez AF, Lindsell CJ. The future of clinical trials: artificial to augmented to applied intelligence. JAMA, 2023, 330(21): 2061-2063.
11. Hassani H, Silva ES. The role of ChatGPT in data science: how AI-assisted conversational interfaces are revolutionizing the field. Big Data Cogn Comput, 2023, 7(2): 62.
12. Sanmarchi F, Bucci A, Nuzzolese AG, et al. A step-by-step researcher's guide to the use of an AI-based transformer in epidemiology: an exploratory analysis of ChatGPT using the STROBE checklist for observational studies. Z Gesundh Wiss, 2023, 5: 1-36.
13. Muka T, Glisic M, Milic J, et al. A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research. Eur J Epidemiol, 2020, 35(1): 49-60.
14. Siddaway AP, Wood AM, Hedges LV. How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annu Rev Psychol, 2019, 70: 747-770.
15. Tawfik GM, Dila KAS, Mohamed MYF, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health, 2019, 47: 46.
16. Mahuli SA, Rai A, Mahuli AV, et al. Application ChatGPT in conducting systematic reviews and meta-analyses. Br Dent J, 2023, 235(2): 90-92.
17. Qureshi R, Shaughnessy D, Gill KAR, et al. Are ChatGPT and large language models "the answer" to bringing us closer to systematic review automation. Syst Rev, 2023, 12(1): 72.
18. van Dijk SHB, Brusse-Keizer MGJ, Bucsán CC, et al. Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open, 2023, 13(7): e072254.
19. Atkinson CF. Cheap, quick, and rigorous: artificial intelligence and the systematic literature review. Social Sci Comp Review, 2023: 08944393231196281.
20. 陈润生. 医疗大数据结合大语言模型的应用展望. 四川大学学报(医学版), 2023, 54(5): 855-856.
21. Scott AM, Glasziou P, Clark J. We extended the 2-week systematic review (2 week SR) methodology to larger, more complex systematic reviews: a case series. J Clin Epidemiol, 2023, 5(157): 112-119.
22. Huang J, Tan M. The role of ChatGPT in scientific communication: writing better scientific review articles. Am J Cancer Res, 2023, 13(4): 1148-1154.
23. Alshami A, Elsayed M, Ali E, et al. Harnessing the power of ChatGPT for automating systematic review process: methodology, case study, limitations, and future directions. Systems, 2023, 11(7): 351.
24. Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc, 2016, 23(1): 193-201.
25. Kim SG. Using ChatGPT for language editing in scientific articles. Maxillofac Plast Reconstr Surg, 2023, 45(1): 13.
26. Gao CA, Howard FM, Markov NS, et al. Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. NPJ Digit Med, 2023, 6(1): 75.
27. Kim JK, Chua M, Rickard M, et al. ChatGPT and large language model (LLM) chatbots: The current state of acceptability and a proposal for guidelines on utilization in academic medicine. J Pediatr Urol, 2023, 19(5): 598-604.
28. Nuzula IF, Amri MM. Will ChatGPT bring a new paradigm to HR world. A critical opinion article. J Manage Stud Devel, 2023, 2(2): 142-161.
29. Peng Y, Rousseau JF, Shortliffe EH, et al. AI-generated text may have a role in evidence-based medicine. Nat Med, 2023, 29(7): 1593-1594.
30. Higgins JP, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ, 2011, 343: d5928.
31. Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ, 2019, 366: l4898.
32. Schulz KF, Altman DG, Moher D, et al. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ, 2010, 340: c332.
33. Pitre T, Jassal T, Talukdar JR, et al. ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: a methods study. medRxiv preprint, 2023.
34. Roberts RH, Ali SR, Hutchings HA, et al. Comparative study of ChatGPT and human evaluators on the assessment of medical literature according to recognised reporting standards. BMJ Health Care Inform, 2023, 30(1): e100830.
35. Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Clinical practice guidelines we can trust. Washington (DC): National Academies Press, 2011.
36. Lahat A, Shachar E, Avidan B, et al. Evaluating the use of large language model in identifying top research questions in gastroenterology. Sci Rep, 2023, 13(1): 4164.
37. Serj Hunt. Use ChatGPT to find contact details of experts in any topic. Youtube, 2024.
38. The New York Times. Do you have a conflict of interest? This robotic assistant may find it first. The New York Times, 2020.
39. Sohail SS, Madsen DØ, Himeur Y, et al. Using ChatGPT to navigate ambivalent and contradictory research findings on artificial intelligence. Front Artif Intell, 2023, 6: 1195797.
40. Hamed E, Eid A, Alberry M. Exploring ChatGPT's potential in facilitating adaptation of clinical guidelines: a case study of diabetic ketoacidosis guidelines. Cureus, 2023, 15(5): e38784.
41. Haverkamp W, Tennenbaum J, Strodthoff N. ChatGPT fails the test of evidence-based medicine. Eur Heart J Digit Health, 2023, 4(5): 366-367.
42. Gaggioli A. Ethics: disclose use of AI in scientific manuscripts. Nature, 2023, 614(7948): 413.
43. Ali K, Barhom N, Tamimi F, et al. ChatGPT-A double-edged sword for healthcare education. Implications for assessments of dental students. Eur J Dent Educ, 2024, 28(1): 206-211.
44. Palal D, Ghonge S, Jadav V, et al. ChatGPT: a double-edged sword? Health Serv Insights, 2023, 16: 11786329231174338.
45. Shen Y, Heacock L, Elias J, et al. ChatGPT and other large language models are double-edged swords. Radiology, 2023, 307(2): e230163.
46. Cacciamani GE, Collins GS, Gill IS. ChatGPT: standard reporting guidelines for responsible use. Nature, 2023, 618(7964): 238.
47. Huo B, Cacciamani GE, Collins GS, et al. Reporting standards for the use of large language model-linked chatbots for health advice. Nat Med, 2023, 29(12): 2988.
48. Luo X, Estill J, Chen Y. The use of ChatGPT in medical research: do we need a reporting guideline. Int J Surg, 2023, 109(12): 3750-3751.
49. 李佩芳, 陈佳丽, 宁宁, 等. ChatGPT在医学领域的应用进展及思考. 华西医学, 2023, 38(10): 1456-1460.

Next Article
Comprehensive intervention for sarcopenia among older adults: an evidence-based clinical practice guideline

Chinese Journal of Evidence-Based Medicine

The application of large language models in the field of evidence-based medicine

Abstract Full text Figures/Tables Video References Cited by

Next Article

Format

Content