Constructing a real-world evidence evaluation system based on Delphi method_Chinese Journal of Evidence-Based Medicine

Authors：

YAO Zhenge ¹ , ZHOU Jiawei ¹ , LU Mengyi ^1,2 , YOU Dongfang ¹ ,  ZHAO Yang ^1,2,3

1. Department of Biostatistics, School of Public Health, Nanjing Medical University, Nanjing 211166, P. R. China;
2. National Vaccine Innovation Platform, Nanjing Medical University, Nanjing 211166, P. R. China;
3. Jiangsu Key Laboratory of Cancer Biomarkers, Prevention and Treatment, Collaborative Innovation Center for Cancer Personalized Medicine, Nanjing 211166, P. R. China;

Corresponding author：

ZHAO Yang, Email: yzhao@njmu.edu.cn

Keywords：

Real-world evidence; Delphi; Quality evaluation; Identification of bias

DOI：

10.7507/1672-2531.202402025

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective We constructed a real-world evidence evaluation system to provide reference for obtaining high-quality evidence in evidence-based medicine.Methods Through the investigation and analysis of the key factors influencing the real-world research evidence, combined with domestic and foreign literature and evaluation tools, we preliminarily constructed the indicators of the real-world evidence evaluation system, then consulted experts in related fields by the Delphi method, modified and determined the final evaluation indicators. Results The indicators of the final real-world evidence evaluation system included 40 items. The recovery efficiencies of the two rounds of expert consultation were 88.2% and 100%; The expert coordination coefficients were 0.174 (P<0.001) and 0.189 (P<0.001). After the second round of consultation, the mean of Likert scale in the range of 3.73~4.93, and the coefficient of variation varied in the range of 0.05~0.21. Conclusion The real-world evidence evaluation system constructed in this study has certain reliability and scientificity, which can provide a basis and help for the transformation of real-world research into high-quality evidence.

1.	国家药品监督管理局. 真实世界证据支持药物研发与审评的指导原则(试行). 2023.
2.	Wang SV, Schneeweiss S. Emulation of randomized clinical trials with nonrandomized database analyses: results of 32 clinical trials. JAMA, 2023, 329(16): 1376-1385.
3.	Sheldrick RC. Randomized trials vs real-world evidence: how can both inform decision-making. JAMA, 2023, 329(16): 1352-1353.
4.	Roberts MH, Ferguson GT. Real-world evidence: bridging gaps in evidence to guide payer decisions. Pharmacoecon Open, 2021, 5(1): 3-11.
5.	Scavone C, di Mauro G, Mascolo A, et al. The new paradigms in clinical research: from early access programs to the novel therapeutic approaches for unmet medical needs. Front Pharmacol, 2019, 10: 111.
6.	Morales DR, Arlett P. RCTs and real world evidence are complementary, not alternatives. BMJ, 2023, 381: 736.
7.	Purpura CA, Garry EM, Honig N, et al. The role of real-world evidence in fda-approved new drug and biologics license applications. Clin Pharmacol Ther, 2022, 111(1): 135-144.
8.	廖茜雯, 姚晨, 张军, 等. 真实世界数据和证据在我国临床决策中的应用现状. 中国食品药品监管, 2023, (10): 24-35.
9.	陈薇, 方赛男, 刘建平, 等. 国际循证医学证据分级体系的发展与现状. 中国中西医结合杂志, 2017, 37(12): 1413-1419.
10.	Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary. Control Clin Trials, 1996, 17(1): 1-12.
11.	Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ, 2008, 336(7650): 924-926.
12.	曹雪, 孟祥然, 王馨, 等. 真实世界观察性研究的质量评价工具ArRoWS解读. 中国循证医学杂志, 2023, 23(2): 227-232.
13.	Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ, 2016, 355: i4919.
14.	廖星, 章轶立, 谢雁鸣. 真实世界研究标准: RECORD清单和GRACE清单的解读. 中国中药杂志, 2015, 40(24): 4734-4738.
15.	陈英耀, 倪明, 胡献之, 等. 公立医疗机构公益性评价指标筛选—基于德尔菲专家咨询法. 中国卫生政策研究, 2012, 5(1): 6-10.
16.	叶莹, 姬艳芳, 张璐, 等. 运用Likert 5级评分法对免疫规划互联网+培训的效果评价. 河南预防医学杂志, 2019, 30(9): 701-703.
17.	宋辰斐, 薛征, 吴淑艳, 等. 《药物香佩疗法预防小儿反复呼吸道感染治未病实践指南》香佩药物的德尔菲法结果分析. 中华中医药杂志, 2017, 32(1): 139-141.
18.	马雪颜, 于河, 吴力群, 等. 基于德尔菲法的儿童胃肠积热评价量表条目筛选. 中华中医药杂志, 2020, 35(2): 851-854.
19.	肖爽, 朱雪琦, 王彦, 等. 基于德尔菲法的临床科研课题全过程质量控制指标体系构建研究. 中医药管理杂志, 2023, 31(21): 1-6.
20.	Schad F, Thronicke A. Real-world evidence-current developments and perspectives. Int J Environ Res Public Health, 2022, 19(16): 10159.
21.	艾飞玲, 胡葵茹, 石钰霖, 等. 基于纽卡斯尔-渥太华量表对中国吸烟队列研究文献的质量评价. 中华疾病控制杂志, 2021, 25(6): 722-729.
22.	Radenkovic D, Keogh SB, Maruthappu M. Data science in modern evidence-based medicine. J R Soc Med, 2019, 112(12): 493-494.
23.	范美玉. 高质量循证医学证据获取与应用研究. 协和医学杂志, 2023, 14(1): 39-43.
24.	Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence - what is it and what can it tell us. N Engl J Med, 2016, 375(23): 2293-2297.
25.	McNair D, Lumpkin M, Kern S, et al. Use of RWE to inform regulatory, public health policy, and intervention priorities for the developing world. Clin Pharmacol Ther, 2022, 111(1): 44-51.

1. 国家药品监督管理局. 真实世界证据支持药物研发与审评的指导原则(试行). 2023.
2. Wang SV, Schneeweiss S. Emulation of randomized clinical trials with nonrandomized database analyses: results of 32 clinical trials. JAMA, 2023, 329(16): 1376-1385.
3. Sheldrick RC. Randomized trials vs real-world evidence: how can both inform decision-making. JAMA, 2023, 329(16): 1352-1353.
4. Roberts MH, Ferguson GT. Real-world evidence: bridging gaps in evidence to guide payer decisions. Pharmacoecon Open, 2021, 5(1): 3-11.
5. Scavone C, di Mauro G, Mascolo A, et al. The new paradigms in clinical research: from early access programs to the novel therapeutic approaches for unmet medical needs. Front Pharmacol, 2019, 10: 111.
6. Morales DR, Arlett P. RCTs and real world evidence are complementary, not alternatives. BMJ, 2023, 381: 736.
7. Purpura CA, Garry EM, Honig N, et al. The role of real-world evidence in fda-approved new drug and biologics license applications. Clin Pharmacol Ther, 2022, 111(1): 135-144.
8. 廖茜雯, 姚晨, 张军, 等. 真实世界数据和证据在我国临床决策中的应用现状. 中国食品药品监管, 2023, (10): 24-35.
9. 陈薇, 方赛男, 刘建平, 等. 国际循证医学证据分级体系的发展与现状. 中国中西医结合杂志, 2017, 37(12): 1413-1419.
10. Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary. Control Clin Trials, 1996, 17(1): 1-12.
11. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ, 2008, 336(7650): 924-926.
12. 曹雪, 孟祥然, 王馨, 等. 真实世界观察性研究的质量评价工具ArRoWS解读. 中国循证医学杂志, 2023, 23(2): 227-232.
13. Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ, 2016, 355: i4919.
14. 廖星, 章轶立, 谢雁鸣. 真实世界研究标准: RECORD清单和GRACE清单的解读. 中国中药杂志, 2015, 40(24): 4734-4738.
15. 陈英耀, 倪明, 胡献之, 等. 公立医疗机构公益性评价指标筛选—基于德尔菲专家咨询法. 中国卫生政策研究, 2012, 5(1): 6-10.
16. 叶莹, 姬艳芳, 张璐, 等. 运用Likert 5级评分法对免疫规划互联网+培训的效果评价. 河南预防医学杂志, 2019, 30(9): 701-703.
17. 宋辰斐, 薛征, 吴淑艳, 等. 《药物香佩疗法预防小儿反复呼吸道感染治未病实践指南》香佩药物的德尔菲法结果分析. 中华中医药杂志, 2017, 32(1): 139-141.
18. 马雪颜, 于河, 吴力群, 等. 基于德尔菲法的儿童胃肠积热评价量表条目筛选. 中华中医药杂志, 2020, 35(2): 851-854.
19. 肖爽, 朱雪琦, 王彦, 等. 基于德尔菲法的临床科研课题全过程质量控制指标体系构建研究. 中医药管理杂志, 2023, 31(21): 1-6.
20. Schad F, Thronicke A. Real-world evidence-current developments and perspectives. Int J Environ Res Public Health, 2022, 19(16): 10159.
21. 艾飞玲, 胡葵茹, 石钰霖, 等. 基于纽卡斯尔-渥太华量表对中国吸烟队列研究文献的质量评价. 中华疾病控制杂志, 2021, 25(6): 722-729.
22. Radenkovic D, Keogh SB, Maruthappu M. Data science in modern evidence-based medicine. J R Soc Med, 2019, 112(12): 493-494.
23. 范美玉. 高质量循证医学证据获取与应用研究. 协和医学杂志, 2023, 14(1): 39-43.
24. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence - what is it and what can it tell us. N Engl J Med, 2016, 375(23): 2293-2297.
25. McNair D, Lumpkin M, Kern S, et al. Use of RWE to inform regulatory, public health policy, and intervention priorities for the developing world. Clin Pharmacol Ther, 2022, 111(1): 44-51.

Chinese Journal of Evidence-Based Medicine

Latest ArticlesConstructing a real-world evidence evaluation system based on Delphi method

Abstract Full text Figures/Tables Video References Cited by

Format

Content