Ruibin Agent versus mainstream large language models: A comparative study on medical literature comprehension with esophageal cancer as a case study_Chinese Journal of Clinical Thoracic and Cardiovascular Surgery

Authors：

WEN Pinghua ¹ , JIANG Zhijie ¹ , JIANG Huan ¹ , YUAN Xianglei ¹ , ZHOU Yu ² , MA Hu ³ , LU Chao ³ ,  HU Bing ¹

1. Department of Gastroenterology and Hepatology, Digestive Endoscopy Medical Engineering Research Laboratory, West China Hospital, Sichuan University, Chengdu, 610041, P. R. China;
2. Shanghai Huihao Yisheng Information Technology Co., Ltd, Shanghai, 200080, P. R. China;
3. Huawei Technologies Co., Ltd, Chengdu, 610000, P. R. China;

Corresponding author：

HU Bing, Email: hubing@wchscu.edu.cn

Keywords：

Artificial intelligence; large language models; interpretation of medical literature; esophageal neoplasms

DOI：

10.7507/1007-4848.202506081

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective To explore the application value of artificial intelligence in medical research assistance, and analyze the key paths to achieve precise execution of model instructions, improvement of model interpretation completeness, and control of hallucinations. Methods Taking esophageal cancer research as the scenario, five types of literature including research articles, case reports, reviews, editorials, and guidelines were selected for model interpretation tests. The model performance was systematically evaluated from five dimensions: recognition accuracy, format accuracy, instruction execution accuracy, content reliability rate, and content completeness index. The performance differences of Ruibin Agent, GPT-4o, Claude 3.7 Sonnet, DeepSeek V3, and DouBao-pro models in medical literature interpretation tasks were compared. Results A total of 15 studies were included, with 3 studies of each type. The five models collectively conducted 1 875 tests. Due to the poor recognition accuracy of the editorial type, the overall recognition accuracy of Ruibin Agent was significantly lower than other models (92.0% vs. 100.0%, P＜0.001). In terms of format accuracy, Ruibin Agent was significantly better than Claude 3.7 Sonnet (98.7% vs. 92.0%, P=0.002) and GPT-4o (98.7% vs. 78.9%, P＜0.001). In terms of instruction execution accuracy, Ruibin Agent was better than GPT-4o (97.3% vs. 80.0%, P＜0.001). In terms of content reliability rate, Ruibin Agent was significantly lower than Claude 3.7 Sonnet (84.0% vs. 92.0%, P=0.010) and DeepSeek V3 (84.0% vs. 94.7%, P＜0.001). In terms of content completeness index, the median scores of Ruibin Agent, GPT-4o, Claude 3.7 Sonnet, DeepSeek V3, and DouBao-pro were 0.71, 0.60, 0.85, 0.74, and 0.77, respectively. Conclusion Ruibin Agent has significant advantages in terms of formatted interpretation of medical literature and instruction execution accuracy. In the future, it is necessary to focus on optimizing the recognition ability of editorial types, strengthening the coverage ability of core elements of various types of literature to improve interpretation completeness, and improving content reliability through optimizing the confidence mechanism to ensure the rigor of medical literature interpretation.

1.	Wang C, Wang F, Li S, et al. Patient triage and guidance in emergency departments using large language models: multimetric study. J Med Internet Res, 2025, 27: e71613.
2.	Yuan XL, Liu W, Lin YX, et al. Effect of an artificial intelligence-assisted system on endoscopic diagnosis of superficial oesophageal squamous cell carcinoma and precancerous lesions: a multicentre, tandem, double-blind, randomised controlled trial. Lancet Gastroenterol Hepatol, 2024, 9(1): 34-44.
3.	McDuff D, Schaekermann M, Tu T, et al. Towards accurate differential diagnosis with large language models. Nature, 2025, 642(8067): 451-457.
4.	Ye J, Huang C, Chen Z, et al. A multi-dimensional constraint framework for evaluating and improving instruction following in large language models. arXiv:2505.07591.
5.	Tang L, Sun Z, Idnay B, et al. Evaluating large language models on medical evidence summarization. NPJ Digit Med, 2023, 6(1): 158.
6.	Zhang Y, Li Y, Cui L, et al. Siren's song in the AI ocean: a survey on hallucination in large language models. arXiv:2309.01219.
7.	石锐, 郑兵, 姚巡, 等. "华西黉医"大模型构建与应用. 中国胸心血管外科临床杂志, 2025, 32(5): 587-593.Shi R, Zheng B, Yao X, et al. Construction and application of the "Huaxi Hongyi" large medical model. Chin J Clin Thorac Cardiovasc Surg, 2025, 32(5): 587-593.
8.	周小芹, 刘慧珍, 王婷, 等. 基于大语言模型的临床预测模型研究报告指南(TRIPOD-LLM)解读. 中国胸心血管外科临床杂志, 2025, 32(7): 940-946.Zhou XQ, Liu HZ, Wang T, et al. Interpretation of the TRIPOD-LLM reporting guideline for studies using large language models. Chin J Clin Thorac Cardiovasc Surg, 2025, 32(7): 940-946.
9.	Liang H, Fan JH, Qiao YL. Epidemiology, etiology, and prevention of esophageal squamous cell carcinoma in China. Cancer Biol Med, 2017, 14(1): 33-41.
10.	Horie Y, Yoshio T, Aoyama K, et al. Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks. Gastrointest Endosc, 2019, 89(1): 25-32.
11.	Azad TD, Chaudhuri AA, Fang P, et al. Circulating tumor DNA analysis for detection of minimal residual disease after chemoradiotherapy for localized esophageal cancer. Gastroenterology, 2020, 158(3): 494-505.e6.
12.	Ishihara R, Arima M, Iizuka T, et al. Endoscopic submucosal dissection/endoscopic mucosal resection guidelines for esophageal cancer. Dig Endosc, 2020, 32(4): 452-493.
13.	Shah MA, Kennedy EB, Catenacci DV, et al. Treatment of locally advanced esophageal carcinoma: ASCO guideline. J Clin Oncol, 2020, 38(23): 2677-2694.
14.	Zhou Q, Wei Y, Zhai H, et al. Comorbid early esophageal cancer and Gongylonema pulchrum infection: a case report. BMC Gastroenterol, 2021, 21(1): 305.
15.	Kitagawa Y, Ishihara R, Ishikawa H, et al. Esophageal cancer practice guidelines 2022 edited by the Japan esophageal society: part 1. Esophagus, 2023, 20(3): 343-372.
16.	Liu CQ, Ma YL, Qin Q, et al. Epidemiology of esophageal cancer in 2020 and projections to 2030 and 2040. Thorac Cancer, 2023, 14(1): 3-11.
17.	Deboever N, Jones CM, Yamashita K, et al. Advances in diagnosis and management of cancer of the esophagus. BMJ, 2024, 385: e074962.
18.	Kato K, Machida R, Ito Y, et al. Doublet chemotherapy, triplet chemotherapy, or doublet chemotherapy combined with radiotherapy as neoadjuvant treatment for locally advanced oesophageal cancer (JCOG1109 NExT): a randomised, controlled, open-label, phase 3 trial. Lancet, 2024, 404(10447): 55-66.
19.	Lin YX, Liu W, Yuan XL, et al. Endoscopic submucosal dissection for the treatment of advanced esophageal cancer subsequent to chemotherapy. Gastrointest Endosc, 2024, 99(6): 1048-1049.
20.	Maan ADI, Koch AD. Artificial intelligence assistance during upper endoscopy: a game changer in detection of esophageal squamous cell carcinoma? Chin Clin Oncol, 2024, 13(6): 89.
21.	Mahawongkajit P, Bunprachoen K. Innovative push percutaneous endoscopic gastrostomy by means of gastropexy and Foley catheter in a patient with advanced head and neck and esophageal cancer. Endoscopy, 2024, 56(S01): E428-E429.
22.	Pubu S, Zhang JW, Yang J. Early diagnosis of esophageal cancer: how to put "early detection" into effect? World J Gastrointest Oncol, 2024, 16(8): 3386-3392.
23.	Wang FM, Mo P, Yan X, et al. Present situation and prospect of immunotherapy for unresectable locally advanced esophageal cancer during peri-radiotherapy. World J Gastrointest Oncol, 2024, 16(1): 1-7.
24.	刘军, 赵文哲. 人工智能技术在临床医学领域的应用与实践. 中华医学信息导报, 2025, 40(9): 14.Liu J, Zhao WZ. Application and practice of artificial intelligence technology in clinical medicine. China Med News, 2025, 40(9): 14.
25.	Zhang K, Yang X, Wang Y, et al. Artificial intelligence in drug development. Nat Med, 2025, 31(1): 45-59.
26.	Wang YJ, Yang K, Wen Y, et al. Screening and diagnosis of cardiovascular disease using artificial intelligence-enabled cardiac magnetic resonance imaging. Nat Med, 2024, 30(5): 1471-1480.
27.	韩序, 刘亮, 楼文晖. 生成式人工智能大型语言模型在消化道癌症领域辅助科研创作的现状分析: 基于2024年美国临床肿瘤学会中国学者数据. 中国实用外科杂志, 2024, 44(8): 894-899.Han X, Liu L, Lou WH. A comprehensive analysis of large language models in generative artificial intelligence-assisted research writing: insights from 2024 ASCO gastrointestinal oncology data by Chinese scholars. Chin J Pract Surg, 2024, 44(8): 894-899.
28.	Bao T, Zhang H, Zhang C. Enhancing abstractive summarization of scientific papers using structure information. Expert Syst Appl, 2025, 261: 125529.
29.	Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc, 2021, 3(1): 2.
30.	Moradi M, Samwald M. Improving the robustness and accuracy of biomedical language models through adversarial training. J Biomed Inform, 2022, 132: 104114.
31.	Liang X, Song S, Zheng Z, et al. Internal consistency and self-feedback in large language models: a survey. arXiv:2407.14507.
32.	刘泽垣, 王鹏江, 宋晓斌, 等. 大语言模型的幻觉问题研究综述. 软件学报, 2025, 36(3): 1152-1185.Liu ZY, Wang PJ, Song XB, et al. Survey on hallucinations in large language models. J Softw, 2025, 36(3): 1152-1185.
33.	Hu M, He B, Wang Y, et al. Mitigating large language model hallucination with faithful finetuning. arXiv:2406.11267.
34.	Xu N, Ma X. DecoPrompt: decoding prompts reduces hallucinations when large language models meet false premises. arXiv:2411.07457.
35.	Fadeeva E, Rubashevskii A, Shelmanov A, et al. Fact-checking the output of large language models via token-level uncertainty quantification. arXiv:2403.04696.

1. Wang C, Wang F, Li S, et al. Patient triage and guidance in emergency departments using large language models: multimetric study. J Med Internet Res, 2025, 27: e71613.
2. Yuan XL, Liu W, Lin YX, et al. Effect of an artificial intelligence-assisted system on endoscopic diagnosis of superficial oesophageal squamous cell carcinoma and precancerous lesions: a multicentre, tandem, double-blind, randomised controlled trial. Lancet Gastroenterol Hepatol, 2024, 9(1): 34-44.
3. McDuff D, Schaekermann M, Tu T, et al. Towards accurate differential diagnosis with large language models. Nature, 2025, 642(8067): 451-457.
4. Ye J, Huang C, Chen Z, et al. A multi-dimensional constraint framework for evaluating and improving instruction following in large language models. arXiv:2505.07591.
5. Tang L, Sun Z, Idnay B, et al. Evaluating large language models on medical evidence summarization. NPJ Digit Med, 2023, 6(1): 158.
6. Zhang Y, Li Y, Cui L, et al. Siren's song in the AI ocean: a survey on hallucination in large language models. arXiv:2309.01219.
7. 石锐, 郑兵, 姚巡, 等. "华西黉医"大模型构建与应用. 中国胸心血管外科临床杂志, 2025, 32(5): 587-593.Shi R, Zheng B, Yao X, et al. Construction and application of the "Huaxi Hongyi" large medical model. Chin J Clin Thorac Cardiovasc Surg, 2025, 32(5): 587-593.
8. 周小芹, 刘慧珍, 王婷, 等. 基于大语言模型的临床预测模型研究报告指南(TRIPOD-LLM)解读. 中国胸心血管外科临床杂志, 2025, 32(7): 940-946.Zhou XQ, Liu HZ, Wang T, et al. Interpretation of the TRIPOD-LLM reporting guideline for studies using large language models. Chin J Clin Thorac Cardiovasc Surg, 2025, 32(7): 940-946.
9. Liang H, Fan JH, Qiao YL. Epidemiology, etiology, and prevention of esophageal squamous cell carcinoma in China. Cancer Biol Med, 2017, 14(1): 33-41.
10. Horie Y, Yoshio T, Aoyama K, et al. Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks. Gastrointest Endosc, 2019, 89(1): 25-32.
11. Azad TD, Chaudhuri AA, Fang P, et al. Circulating tumor DNA analysis for detection of minimal residual disease after chemoradiotherapy for localized esophageal cancer. Gastroenterology, 2020, 158(3): 494-505.e6.
12. Ishihara R, Arima M, Iizuka T, et al. Endoscopic submucosal dissection/endoscopic mucosal resection guidelines for esophageal cancer. Dig Endosc, 2020, 32(4): 452-493.
13. Shah MA, Kennedy EB, Catenacci DV, et al. Treatment of locally advanced esophageal carcinoma: ASCO guideline. J Clin Oncol, 2020, 38(23): 2677-2694.
14. Zhou Q, Wei Y, Zhai H, et al. Comorbid early esophageal cancer and Gongylonema pulchrum infection: a case report. BMC Gastroenterol, 2021, 21(1): 305.
15. Kitagawa Y, Ishihara R, Ishikawa H, et al. Esophageal cancer practice guidelines 2022 edited by the Japan esophageal society: part 1. Esophagus, 2023, 20(3): 343-372.
16. Liu CQ, Ma YL, Qin Q, et al. Epidemiology of esophageal cancer in 2020 and projections to 2030 and 2040. Thorac Cancer, 2023, 14(1): 3-11.
17. Deboever N, Jones CM, Yamashita K, et al. Advances in diagnosis and management of cancer of the esophagus. BMJ, 2024, 385: e074962.
18. Kato K, Machida R, Ito Y, et al. Doublet chemotherapy, triplet chemotherapy, or doublet chemotherapy combined with radiotherapy as neoadjuvant treatment for locally advanced oesophageal cancer (JCOG1109 NExT): a randomised, controlled, open-label, phase 3 trial. Lancet, 2024, 404(10447): 55-66.
19. Lin YX, Liu W, Yuan XL, et al. Endoscopic submucosal dissection for the treatment of advanced esophageal cancer subsequent to chemotherapy. Gastrointest Endosc, 2024, 99(6): 1048-1049.
20. Maan ADI, Koch AD. Artificial intelligence assistance during upper endoscopy: a game changer in detection of esophageal squamous cell carcinoma? Chin Clin Oncol, 2024, 13(6): 89.
21. Mahawongkajit P, Bunprachoen K. Innovative push percutaneous endoscopic gastrostomy by means of gastropexy and Foley catheter in a patient with advanced head and neck and esophageal cancer. Endoscopy, 2024, 56(S01): E428-E429.
22. Pubu S, Zhang JW, Yang J. Early diagnosis of esophageal cancer: how to put "early detection" into effect? World J Gastrointest Oncol, 2024, 16(8): 3386-3392.
23. Wang FM, Mo P, Yan X, et al. Present situation and prospect of immunotherapy for unresectable locally advanced esophageal cancer during peri-radiotherapy. World J Gastrointest Oncol, 2024, 16(1): 1-7.
24. 刘军, 赵文哲. 人工智能技术在临床医学领域的应用与实践. 中华医学信息导报, 2025, 40(9): 14.Liu J, Zhao WZ. Application and practice of artificial intelligence technology in clinical medicine. China Med News, 2025, 40(9): 14.
25. Zhang K, Yang X, Wang Y, et al. Artificial intelligence in drug development. Nat Med, 2025, 31(1): 45-59.
26. Wang YJ, Yang K, Wen Y, et al. Screening and diagnosis of cardiovascular disease using artificial intelligence-enabled cardiac magnetic resonance imaging. Nat Med, 2024, 30(5): 1471-1480.
27. 韩序, 刘亮, 楼文晖. 生成式人工智能大型语言模型在消化道癌症领域辅助科研创作的现状分析: 基于2024年美国临床肿瘤学会中国学者数据. 中国实用外科杂志, 2024, 44(8): 894-899.Han X, Liu L, Lou WH. A comprehensive analysis of large language models in generative artificial intelligence-assisted research writing: insights from 2024 ASCO gastrointestinal oncology data by Chinese scholars. Chin J Pract Surg, 2024, 44(8): 894-899.
28. Bao T, Zhang H, Zhang C. Enhancing abstractive summarization of scientific papers using structure information. Expert Syst Appl, 2025, 261: 125529.
29. Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc, 2021, 3(1): 2.
30. Moradi M, Samwald M. Improving the robustness and accuracy of biomedical language models through adversarial training. J Biomed Inform, 2022, 132: 104114.
31. Liang X, Song S, Zheng Z, et al. Internal consistency and self-feedback in large language models: a survey. arXiv:2407.14507.
32. 刘泽垣, 王鹏江, 宋晓斌, 等. 大语言模型的幻觉问题研究综述. 软件学报, 2025, 36(3): 1152-1185.Liu ZY, Wang PJ, Song XB, et al. Survey on hallucinations in large language models. J Softw, 2025, 36(3): 1152-1185.
33. Hu M, He B, Wang Y, et al. Mitigating large language model hallucination with faithful finetuning. arXiv:2406.11267.
34. Xu N, Ma X. DecoPrompt: decoding prompts reduces hallucinations when large language models meet false premises. arXiv:2411.07457.
35. Fadeeva E, Rubashevskii A, Shelmanov A, et al. Fact-checking the output of large language models via token-level uncertainty quantification. arXiv:2403.04696.

Chinese Journal of Clinical Thoracic and Cardiovascular Surgery

Latest ArticlesRuibin Agent versus mainstream large language models: A comparative study on medical literature comprehension with esophageal cancer as a case study

Abstract Full text Figures/Tables Video References Cited by

Format

Content