Statistical methods in multi-reader multi-case design diagnostic accuracy studies_Chinese Journal of Evidence-Based Medicine

Authors：

HE Qian ¹ , PAN Zhemin ² , XIANG Man ² , WAN Huiqin ² ,  QIN Yingyi ¹ ,  HE Jia ^1,2

1. Department of Military Health Statistics, Naval Medical University, Shanghai 200433, P. R. China;
2. Tongji University School of Medicine, Shanghai 200092, P. R. China;

Corresponding author：

QIN Yingyi, Email: yingyi_qin@163.com; HE Jia, Email: hejia63@yeah.net

Keywords：

Multi-reader multi-case (MRMC); Medical imaging; Diagnostic test; Accuracy

DOI：

10.7507/1672-2531.202312140

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective To elaborate on the statistical analysis methods for evaluating the accuracy of imaging diagnostic tests in a multiple-reader multiple-case (MRMC) design through formula derivation and real cases. Methods This study consisted of two parts: theoretical derivation and a real case study. The theoretical part discussed in detail the principles and procedures of MRMC statistical analysis methods, particularly the Obuchowski-Rockette (OR) and Dorfman-Berbaum-Metz (DBM) methods. The real case included 100 subjects, of whom 67 had disease. Four readers interpreted all the cases based on both traditional film imaging methods and digital imaging methods. OR and DBM methods were employed for data analysis. Results The real case showed that the OR and DBM methods had a high degree of consistency, with only slight differences in the confidence intervals. Conclusion It is recommended to use the OR and DBM methods for the statistical analysis of imaging diagnostic test accuracy, ensuring that the impact of reader factors on the evaluation results is fully considered. The results from the OR and DBM methods are relatively similar; when applying these methods in practice, one should consider the specific characteristics of the data and the research design to choose the appropriate analysis method. Besides, there are still challenges when applying the OR and DBM methods, such as software implementation and missing data handling, which require further exploration.

Citation： HE Qian, PAN Zhemin, XIANG Man, WAN Huiqin, QIN Yingyi, HE Jia. Statistical methods in multi-reader multi-case design diagnostic accuracy studies. Chinese Journal of Evidence-Based Medicine, 2024, 24(9): 1085-1093. doi: 10.7507/1672-2531.202312140 Copy

1.	Chakraborty DP. Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol, 2006, 13(10): 1187-1193.
2.	Shile PE, Pilgram TK. Variability in the interpretation of screening mammograms by US radiologists. Acad Radiol, 1996, 3(10): 879-881.
3.	Wagner RF, Metz CE, Campbell G. Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol, 2007, 14(6): 723-748.
4.	Zhou XH, Obuchowski NA, Mcclish DK. Statistical methods in diagnostic medicine (2nd Edition). New York: John Wiley & Sons, Inc. , 2011.
5.	Wang L, Wang H, Xia C, et al. Toward standardized premarket evaluation of computer aided diagnosis/detection products: insights from FDA-approved products. Expert Rev Med Devices, 2020, 17(9): 899-918.
6.	FDA. Clinical performance assessment: considerations for computer-assisted detection devices applied to radiology images and radiology device data in premarket notification (510(k)) submissions: guidance for industry and Food and Drug Administration staff. 2022.
7.	Gallas BD, Chan HP, D'Orsi CJ, et al. Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol, 2012, 19(4): 463-477.
8.	医疗器械技术审评中心. 深度学习辅助决策医疗器械软件审评要点. 2023.
9.	国家药品监督管理局. 乳腺X射线系统注册技术审查指导原则. 2023.
10.	尚美霞, 姚晨, 阎小妍, 等. 影像诊断试验中多阅片者研究的设计与分析. 中国卫生统计, 2014, 31(2): 331-335.
11.	Obuchowski NA, Bullen J. Multireader diagnostic accuracy imaging studies: fundamentals of design and analysis. Radiology, 2022, 303(1): 26-34.
12.	Dendumrongsup T, Plumb AA, Halligan S, et al. Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: systematic review with a focus on quality of data reporting. PLoS One, 2014, 9(12): e116018.
13.	Obuchowski NA, Rockette HE. Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: an anova approach with dependent observations. Commun Stat Simul Comput, 1995, 24(2): 285-308.
14.	Chakraborty DP. Observer performance methods for diagnostic imaging: foundations, modeling, and applications with r-based examples. Boca Raton: CRC Press, 2017.
15.	Efron B, Tibshirani RJ. An introduction to the bootstrap. Monogr Stat Appl Probab, 1993, 57: 158.
16.	Delong ER, Delong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 1988, 44(3): 837-845.
17.	Pavur R, Nath R. Exact F tests in an ANOVA procedure for dependent observations. Multivariate Behav Res, 1984, 19(4): 408-420.
18.	Hillis SL, Obuchowski NA, Schartz KM, et al. A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data. Stat Med, 2005, 24(10): 1579-1607.
19.	Hillis SL, Berbaum KS, Metz CE. Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Acad Radiol, 2008, 15(5): 647-661.
20.	Hillis SL. A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Stat Med, 2007, 26(3): 596-619.
21.	Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Invest Radiol, 1992, 27(9): 723-731.
22.	Obuchowski NA, Beiden SV, Berbaum KS, et al. Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol, 2004, 11(9): 980-995.
23.	Hillis SL. A marginal-mean ANOVA approach for analyzing multireader multicase radiological imaging data. Stat Med, 2014, 33(2): 330-360.
24.	Satterthwaite FE. Synthesis of variance. Psychometrika, 1941, 6(5): 309-316.
25.	Satterthwaite FE. An approximate distribution of estimates of variance components. Biometrics, 1946, 2(6): 110-114.
26.	Berbaum KS. God, like the Devil, is in the details. Acad Radiol, 2006, 13(11): 1311-1316.
27.	Dorfman DD, Berbaum KS, Lenth RV, et al. Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design. Acad Radiol, 1998, 5(9): 591-602.
28.	Roe CA, Metz CE. Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation. Acad Radiol, 1997, 4(4): 298-303.
29.	Gaylor DW, Hopper FN. Estimating the degrees of freedom for linear combinations of mean squares by Satterthwaite's formula. Technometrics, 1969, 11(4): 691-706.
30.	Hillis SL, Berbaum KS. Power estimation for the Dorfman-Berbaum-Metz method. Acad Radiol, 2004, 11(11): 1260-1273.
31.	Franken EA, Berbaum KS, Marley SM, et al. Evaluation of a digital workstation for interpreting neonatal examinations. A receiver operating characteristic study. Invest Radiol, 1992, 27(9): 732-737.
32.	Hillis SL. OR-DBM MRMC 2.51. 2023.
33.	Smith BJ, Hillis SL. Multi-reader multi-case analysis of variance software for diagnostic performance comparison of imaging modalities. Proc SPIE Int Soc Opt Eng, 2020, 11316: 113160K.
34.	RJafroc: artificial intelligence systems and observer performance. 2023.
35.	尚美霞, 姚晨, 康晓平, 等. MRMC方差分析在影像诊断试验多阅片者多病例研究设计中的应用. 中国卫生统计, 2017, 34(5): 705-709.
36.	Little R, Rubin DB. Statistical analysis with missing data. New York: John Wiley & Sons, Inc. , 2014.
37.	Pedersen AB, Mikkelsen EM, Cronin-Fenton D, et al. Missing data and multiple imputation in clinical epidemiological research. Clin Epidemiol, 2017, 9: 157-166.
38.	Rubin DB. Formalizing subjective notions about the effect of nonrespondents in sample surveys. J Am Stat Assoc, 1977, 72(359): 538-543.

1. Chakraborty DP. Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol, 2006, 13(10): 1187-1193.
2. Shile PE, Pilgram TK. Variability in the interpretation of screening mammograms by US radiologists. Acad Radiol, 1996, 3(10): 879-881.
3. Wagner RF, Metz CE, Campbell G. Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol, 2007, 14(6): 723-748.
4. Zhou XH, Obuchowski NA, Mcclish DK. Statistical methods in diagnostic medicine (2nd Edition). New York: John Wiley & Sons, Inc. , 2011.
5. Wang L, Wang H, Xia C, et al. Toward standardized premarket evaluation of computer aided diagnosis/detection products: insights from FDA-approved products. Expert Rev Med Devices, 2020, 17(9): 899-918.
6. FDA. Clinical performance assessment: considerations for computer-assisted detection devices applied to radiology images and radiology device data in premarket notification (510(k)) submissions: guidance for industry and Food and Drug Administration staff. 2022.
7. Gallas BD, Chan HP, D'Orsi CJ, et al. Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol, 2012, 19(4): 463-477.
8. 医疗器械技术审评中心. 深度学习辅助决策医疗器械软件审评要点. 2023.
9. 国家药品监督管理局. 乳腺X射线系统注册技术审查指导原则. 2023.
10. 尚美霞, 姚晨, 阎小妍, 等. 影像诊断试验中多阅片者研究的设计与分析. 中国卫生统计, 2014, 31(2): 331-335.
11. Obuchowski NA, Bullen J. Multireader diagnostic accuracy imaging studies: fundamentals of design and analysis. Radiology, 2022, 303(1): 26-34.
12. Dendumrongsup T, Plumb AA, Halligan S, et al. Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: systematic review with a focus on quality of data reporting. PLoS One, 2014, 9(12): e116018.
13. Obuchowski NA, Rockette HE. Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: an anova approach with dependent observations. Commun Stat Simul Comput, 1995, 24(2): 285-308.
14. Chakraborty DP. Observer performance methods for diagnostic imaging: foundations, modeling, and applications with r-based examples. Boca Raton: CRC Press, 2017.
15. Efron B, Tibshirani RJ. An introduction to the bootstrap. Monogr Stat Appl Probab, 1993, 57: 158.
16. Delong ER, Delong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 1988, 44(3): 837-845.
17. Pavur R, Nath R. Exact F tests in an ANOVA procedure for dependent observations. Multivariate Behav Res, 1984, 19(4): 408-420.
18. Hillis SL, Obuchowski NA, Schartz KM, et al. A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data. Stat Med, 2005, 24(10): 1579-1607.
19. Hillis SL, Berbaum KS, Metz CE. Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Acad Radiol, 2008, 15(5): 647-661.
20. Hillis SL. A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Stat Med, 2007, 26(3): 596-619.
21. Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Invest Radiol, 1992, 27(9): 723-731.
22. Obuchowski NA, Beiden SV, Berbaum KS, et al. Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol, 2004, 11(9): 980-995.
23. Hillis SL. A marginal-mean ANOVA approach for analyzing multireader multicase radiological imaging data. Stat Med, 2014, 33(2): 330-360.
24. Satterthwaite FE. Synthesis of variance. Psychometrika, 1941, 6(5): 309-316.
25. Satterthwaite FE. An approximate distribution of estimates of variance components. Biometrics, 1946, 2(6): 110-114.
26. Berbaum KS. God, like the Devil, is in the details. Acad Radiol, 2006, 13(11): 1311-1316.
27. Dorfman DD, Berbaum KS, Lenth RV, et al. Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design. Acad Radiol, 1998, 5(9): 591-602.
28. Roe CA, Metz CE. Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation. Acad Radiol, 1997, 4(4): 298-303.
29. Gaylor DW, Hopper FN. Estimating the degrees of freedom for linear combinations of mean squares by Satterthwaite's formula. Technometrics, 1969, 11(4): 691-706.
30. Hillis SL, Berbaum KS. Power estimation for the Dorfman-Berbaum-Metz method. Acad Radiol, 2004, 11(11): 1260-1273.
31. Franken EA, Berbaum KS, Marley SM, et al. Evaluation of a digital workstation for interpreting neonatal examinations. A receiver operating characteristic study. Invest Radiol, 1992, 27(9): 732-737.
32. Hillis SL. OR-DBM MRMC 2.51. 2023.
33. Smith BJ, Hillis SL. Multi-reader multi-case analysis of variance software for diagnostic performance comparison of imaging modalities. Proc SPIE Int Soc Opt Eng, 2020, 11316: 113160K.
34. RJafroc: artificial intelligence systems and observer performance. 2023.
35. 尚美霞, 姚晨, 康晓平, 等. MRMC方差分析在影像诊断试验多阅片者多病例研究设计中的应用. 中国卫生统计, 2017, 34(5): 705-709.
36. Little R, Rubin DB. Statistical analysis with missing data. New York: John Wiley & Sons, Inc. , 2014.
37. Pedersen AB, Mikkelsen EM, Cronin-Fenton D, et al. Missing data and multiple imputation in clinical epidemiological research. Clin Epidemiol, 2017, 9: 157-166.
38. Rubin DB. Formalizing subjective notions about the effect of nonrespondents in sample surveys. J Am Stat Assoc, 1977, 72(359): 538-543.

Chinese Journal of Evidence-Based Medicine

Statistical methods in multi-reader multi-case design diagnostic accuracy studies

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content