Multimodal deep learning model for staging diabetic retinopathy based on ultra-widefield fluorescence angiography_Chinese Journal of Ocular Fundus Diseases

Authors：

Fan Wen ¹ , Wang Xiaoling ² , Ma Xiao ³ , Yuan Songtao ¹ , Chen Changzheng ² ,  Ji Zexuan ³

1. Department of Ophthalmology, The First Affliated Hospital with Nanjing Medical University, Nanjing 210029, China;
2. Eye Center, Renmin Hospital of Wuhan University, Wuhan 430060, China;
3. School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China;
Chen Changzheng and Ji Zexuan are contributed equally to the article;

Corresponding author：

Ji Zexuan, Email: jizexuan@njust.edu.cn

Keywords：

Artificial intelligence; Neural networks (computer); Fluorescein angiography; Diabetic retinopathy; Lesion classification; Lesion location

DOI：

10.3760/cma.j.cn511434-20211231-00736

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective To apply the multi-modal deep learning model to automatically classify the ultra-widefield fluorescein angiography (UWFA) images of diabetic retinopathy (DR). Methods A retrospective study. From 2015 to 2020, 798 images of 297 DR patients with 399 eyes who were admitted to Eye Center of Renmin Hospital of Wuhan University and were examined by UWFA were used as the training set and test set of the model. Among them, 119, 171, and 109 eyes had no retinopathy, non-proliferative DR (NPDR), and proliferative DR (PDR), respectively. Localization and assessment of fluorescein leakage and non-perfusion regions in early and late orthotopic images of UWFA in DR-affected eyes by jointly optimizing CycleGAN and a convolutional neural network (CNN) classifier, an image-level supervised deep learning model. The abnormal images with lesions were converted into normal images with lesions removed using the improved CycleGAN, and the difference images containing the lesion areas were obtained; the difference images were classified by the CNN classifier to obtain the prediction results. A five-fold cross-test was used to evaluate the classification accuracy of the model. Quantitative analysis of the marker area displayed by the differential images was performed to observe the correlation between the ischemia index and leakage index and the severity of DR. Results The generated fake normal image basically removed all the lesion areas while retaining the normal vascular structure; the difference images intuitively revealed the distribution of biomarkers; the heat icon showed the leakage area, and the location was basically the same as the lesion area in the original image. The results of the five-fold cross-check showed that the average classification accuracy of the model was 0.983. Further quantitative analysis of the marker area showed that the ischemia index and leakage index were significantly positively correlated with the severity of DR (β=6.088, 10.850; P＜0.001). Conclusion The constructed multimodal joint optimization model can accurately classify NPDR and PDR and precisely locate potential biomarkers.

Citation： Fan Wen, Wang Xiaoling, Ma Xiao, Yuan Songtao, Chen Changzheng, Ji Zexuan. Multimodal deep learning model for staging diabetic retinopathy based on ultra-widefield fluorescence angiography. Chinese Journal of Ocular Fundus Diseases, 2022, 38(2): 139-145. doi: 10.3760/cma.j.cn511434-20211231-00736 Copy

1.	Wessel MM, Aaker GD, Parlitsis G, et al. Ultra-wide-field angiography improves the detection and classification of diabetic retinopathy[J]. Retina, 2012, 32(4): 785-791. DOI: 10.1097/IAE.0b013e3182278b64.
2.	Cai S, Liu TYA. The role of ultra-widefield fundus imaging and fluorescein angiography in diagnosis and treatment of diabetic retinopathy[J]. Curr Diab Rep, 2021, 21(9): 30. DOI: 10.1007/s11892-021-01398-0.
3.	Ehlers JP, Jiang AC, Boss JD, et al. Quantitative ultra-widefield angiography and diabetic retinopathy severity: an assessment of panretinal leakage index, ischemic index and microaneurysm count[J]. Ophthalmology, 2019, 126(11): 1527-1532. DOI: 10.1016/j.ophtha.2019.05.034.
4.	Jiang A, Srivastava S, Figueiredo N, et al. Repeatability of automated leakage quantification and microaneurysm identification utilising an analysis platform for ultra-widefield fluorescein angiography[J]. Br J Ophthalmol, 2020, 104(4): 500-503. DOI: 10.1136/bjophthalmol-2019-314416.
5.	Jiang AC, Srivastava SK, Hu M, et al. Quantitative ultra-widefield angiographic features and associations with diabetic macular edema[J]. Ophthalmol Retina, 2020, 4(1): 49-56. DOI: 10.1016/j.oret.2019.08.008.
6.	Aiello LP, Odia I, Glassman AR, et al. Comparison of early treatment diabetic retinopathy study standard 7-field imaging with ultrawide-field imaging for determining severity of diabetic retinopathy[J]. JAMA Ophthalmol, 2019, 137(1): 65-73. DOI: 10.1001/jamaophthalmol.2018.4982.
7.	Solomon SD, Goldberg MF. ETDRS grading of diabetic retinopathy: still the gold standard?[J]. Ophthalmic Res, 2019, 62(4): 190-195. DOI: 10.1159/000501372.
8.	Zhang Z, Ji Z, Chen Q, et al. Joint optimization of cycleGAN and CNN classifier for detection and localization of retinal pathologies on color fundus photographs[J]. IEEE J Biomed Health Inform, 2022, 26(1): 115-126. DOI: 10.1109/jbhi.2021.3092339.
9.	Early Treatment Diabetic Retinopathy Study Research Group. Grading diabetic retinopathy from stereoscopic color fundus photographs-an extension of the modified airlie house classification: ETDRS report number 10[J]. Ophthalmology, 2020, 127(4): S99-S119. DOI: 10.1016/j.ophtha.2020.01.030.
10.	Nicholson L, Vazquez-Alfageme C, Ramu J, et al. Validation of concentric rings method as a topographic measure of retinal nonperfusion in ultra-widefield fluorescein angiography[J]. Am J Ophthalmol, 2015, 160(6): 1217-1225. DOI: 10.1016/j.ajo.2015.09.003.
11.	Ehlers JP, Wang K, Vasanji A, et al. Automated quantitative characterisation of retinal vascular leakage and microaneurysms in ultra-widefield fluorescein angiography[J]. Br J Ophthalmol, 2017, 101(6): 696-699. DOI: 10.1136/bjophthalmol-2016-310047.
12.	Ding L, Bawany MH, Kuriyan AE, et al. A novel deep learning pipeline for retinal vessel detection in fluorescein angiography[J]. IEEE Trans Image Process, 2020, 29: 6561-6573. DOI: 10.1109/TIP.2020.2991530.
13.	Pan X, Jin K, Cao J, et al. Multi-label classification of retinal lesions in diabetic retinopathy for automatic analysis of fundus fluorescein angiography based on deep learning[J]. Graefe's Arch Clin Exp Ophthalmol, 2020, 258(4): 779-785. DOI: 10.1007/s00417-019-04575-w.
14.	Nunez do Rio JM, Sen P, Rasheed R, et al. Deep learning-based segmentation and quantification of retinal capillary non-perfusion on ultra-wide-field retinal fluorescein angiography[J/OL]. J Clin Med, 2020, 9(8): 2537[2020-08-06]. https://pubmed.ncbi.nlm.nih.gov/32781564/. DOI: 10.3390/jcm9082537.
15.	Ye Y, Mao J, Liu L, et al. Automatic diagnosis of familial exudative vitreoretinopathy using a fusion neural network for wide-angle retinal images[J]. IEEE Access, 2020, 8: 162-173. DOI: 10.1109/ACCESS.2019.2961418.
16.	Li Z, Guo C, Nie D, et al. Deep learning for detecting retinal detachment and discerning macular status using ultra-widefield fundus images[J]. Commun Biol, 2020, 3(1): 15. DOI: 10.1038/s42003-019-0730-x.
17.	Schlegl T, Seeböck P, Waldstein SM, et al. f-AnoGAN: fast unsupervised anomaly detection with generative adversarial networks[J]. Med Image Anal, 2019, 54: 30-44. DOI: 10.1016/j.media.2019.01.010.
18.	Siddiquee MMR, Zhou Z, Tajbakhsh N, et al. Learning fixed points in generative adversarial networks: from image-to-image translation to disease detection and localization[J]. Proc IEEE Int Conf Comput Vis, 2019, 2019: 191-200. DOI: 10.1109/iccv.2019.00028.
19.	Zhang R, Tan S, Wang R, et al. Biomarker localization by combining CNN classifier and generative adversarial network[C/OL]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2019, Shenzhen, 2019[2019-10-13]. https://doi.org/10.1007/978-3-030-32239-7_24).
20.	Zhu JY, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, 2017: 2223-2232.

1. Wessel MM, Aaker GD, Parlitsis G, et al. Ultra-wide-field angiography improves the detection and classification of diabetic retinopathy[J]. Retina, 2012, 32(4): 785-791. DOI: 10.1097/IAE.0b013e3182278b64.
2. Cai S, Liu TYA. The role of ultra-widefield fundus imaging and fluorescein angiography in diagnosis and treatment of diabetic retinopathy[J]. Curr Diab Rep, 2021, 21(9): 30. DOI: 10.1007/s11892-021-01398-0.
3. Ehlers JP, Jiang AC, Boss JD, et al. Quantitative ultra-widefield angiography and diabetic retinopathy severity: an assessment of panretinal leakage index, ischemic index and microaneurysm count[J]. Ophthalmology, 2019, 126(11): 1527-1532. DOI: 10.1016/j.ophtha.2019.05.034.
4. Jiang A, Srivastava S, Figueiredo N, et al. Repeatability of automated leakage quantification and microaneurysm identification utilising an analysis platform for ultra-widefield fluorescein angiography[J]. Br J Ophthalmol, 2020, 104(4): 500-503. DOI: 10.1136/bjophthalmol-2019-314416.
5. Jiang AC, Srivastava SK, Hu M, et al. Quantitative ultra-widefield angiographic features and associations with diabetic macular edema[J]. Ophthalmol Retina, 2020, 4(1): 49-56. DOI: 10.1016/j.oret.2019.08.008.
6. Aiello LP, Odia I, Glassman AR, et al. Comparison of early treatment diabetic retinopathy study standard 7-field imaging with ultrawide-field imaging for determining severity of diabetic retinopathy[J]. JAMA Ophthalmol, 2019, 137(1): 65-73. DOI: 10.1001/jamaophthalmol.2018.4982.
7. Solomon SD, Goldberg MF. ETDRS grading of diabetic retinopathy: still the gold standard?[J]. Ophthalmic Res, 2019, 62(4): 190-195. DOI: 10.1159/000501372.
8. Zhang Z, Ji Z, Chen Q, et al. Joint optimization of cycleGAN and CNN classifier for detection and localization of retinal pathologies on color fundus photographs[J]. IEEE J Biomed Health Inform, 2022, 26(1): 115-126. DOI: 10.1109/jbhi.2021.3092339.
9. Early Treatment Diabetic Retinopathy Study Research Group. Grading diabetic retinopathy from stereoscopic color fundus photographs-an extension of the modified airlie house classification: ETDRS report number 10[J]. Ophthalmology, 2020, 127(4): S99-S119. DOI: 10.1016/j.ophtha.2020.01.030.
10. Nicholson L, Vazquez-Alfageme C, Ramu J, et al. Validation of concentric rings method as a topographic measure of retinal nonperfusion in ultra-widefield fluorescein angiography[J]. Am J Ophthalmol, 2015, 160(6): 1217-1225. DOI: 10.1016/j.ajo.2015.09.003.
11. Ehlers JP, Wang K, Vasanji A, et al. Automated quantitative characterisation of retinal vascular leakage and microaneurysms in ultra-widefield fluorescein angiography[J]. Br J Ophthalmol, 2017, 101(6): 696-699. DOI: 10.1136/bjophthalmol-2016-310047.
12. Ding L, Bawany MH, Kuriyan AE, et al. A novel deep learning pipeline for retinal vessel detection in fluorescein angiography[J]. IEEE Trans Image Process, 2020, 29: 6561-6573. DOI: 10.1109/TIP.2020.2991530.
13. Pan X, Jin K, Cao J, et al. Multi-label classification of retinal lesions in diabetic retinopathy for automatic analysis of fundus fluorescein angiography based on deep learning[J]. Graefe's Arch Clin Exp Ophthalmol, 2020, 258(4): 779-785. DOI: 10.1007/s00417-019-04575-w.
14. Nunez do Rio JM, Sen P, Rasheed R, et al. Deep learning-based segmentation and quantification of retinal capillary non-perfusion on ultra-wide-field retinal fluorescein angiography[J/OL]. J Clin Med, 2020, 9(8): 2537[2020-08-06]. https://pubmed.ncbi.nlm.nih.gov/32781564/. DOI: 10.3390/jcm9082537.
15. Ye Y, Mao J, Liu L, et al. Automatic diagnosis of familial exudative vitreoretinopathy using a fusion neural network for wide-angle retinal images[J]. IEEE Access, 2020, 8: 162-173. DOI: 10.1109/ACCESS.2019.2961418.
16. Li Z, Guo C, Nie D, et al. Deep learning for detecting retinal detachment and discerning macular status using ultra-widefield fundus images[J]. Commun Biol, 2020, 3(1): 15. DOI: 10.1038/s42003-019-0730-x.
17. Schlegl T, Seeböck P, Waldstein SM, et al. f-AnoGAN: fast unsupervised anomaly detection with generative adversarial networks[J]. Med Image Anal, 2019, 54: 30-44. DOI: 10.1016/j.media.2019.01.010.
18. Siddiquee MMR, Zhou Z, Tajbakhsh N, et al. Learning fixed points in generative adversarial networks: from image-to-image translation to disease detection and localization[J]. Proc IEEE Int Conf Comput Vis, 2019, 2019: 191-200. DOI: 10.1109/iccv.2019.00028.
19. Zhang R, Tan S, Wang R, et al. Biomarker localization by combining CNN classifier and generative adversarial network[C/OL]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2019, Shenzhen, 2019[2019-10-13]. https://doi.org/10.1007/978-3-030-32239-7_24).
20. Zhu JY, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, 2017: 2223-2232.

Chinese Journal of Ocular Fundus Diseases

Multimodal deep learning model for staging diabetic retinopathy based on ultra-widefield fluorescence angiography

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content