Skin lesion classification with multi-level fusion of Swin-T and ConvNeXt_Journal of Biomedical Engineering

Authors：

WANG Zetong ,  ZHANG Junhua , WANG Xiao

School of Information Science and Engineering, Yunnan University, Kunming 650500, P. R. China;

Corresponding author：

ZHANG Junhua, Email: jhzhang@ynu.edu.cn

Keywords：

Swin-T; ConvNeXt; Multi-level attention mechanisms; Hierarchical inverted residual fusion module; Skin lesion images

DOI：

10.7507/1001-5515.202305025

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Skin cancer is a significant public health issue, and computer-aided diagnosis technology can effectively alleviate this burden. Accurate identification of skin lesion types is crucial when employing computer-aided diagnosis. This study proposes a multi-level attention cascaded fusion model based on Swin-T and ConvNeXt. It employed hierarchical Swin-T and ConvNeXt to extract global and local features, respectively, and introduced residual channel attention and spatial attention modules for further feature extraction. Multi-level attention mechanisms were utilized to process multi-scale global and local features. To address the problem of shallow features being lost due to their distance from the classifier, a hierarchical inverted residual fusion module was proposed to dynamically adjust the extracted feature information. Balanced sampling strategies and focal loss were employed to tackle the issue of imbalanced categories of skin lesions. Experimental testing on the ISIC2018 and ISIC2019 datasets yielded accuracy, precision, recall, and F1-Score of 96.01%, 93.67%, 92.65%, and 93.11%, respectively, and 92.79%, 91.52%, 88.90%, and 90.15%, respectively. Compared to Swin-T, the proposed method achieved an accuracy improvement of 3.60% and 1.66%, and compared to ConvNeXt, it achieved an accuracy improvement of 2.87% and 3.45%. The experiments demonstrate that the proposed method accurately classifies skin lesion images, providing a new solution for skin cancer diagnosis.

Citation： WANG Zetong, ZHANG Junhua, WANG Xiao. Skin lesion classification with multi-level fusion of Swin-T and ConvNeXt. Journal of Biomedical Engineering, 2024, 41(3): 544-551. doi: 10.7507/1001-5515.202305025 Copy

1.	Kumari S, Choudhary P K, Shukla R, et al. Recent advances in nanotechnology based combination drug therapy for skin cancer. J Biomater Sci Polym Ed, 2022, 33(11): 1435-1468.
2.	Steiner A, Binder M, Schemper M, et al. Statistical evaluation of epiluminescence microscopy criteria for melanocytic pigmented skin lesions. J Am Acad Dermatol, 1993, 29(4): 581-588.
3.	Celebi M E, Iyatomi H, Schaefer G, et al. Lesion border detection in dermoscopy images. Comput Med Imaging Graph, 2009, 33(2): 148-153.
4.	Ganster H, Pinz P, Rohrer R, et al. Automated melanoma recognition. IEEE Trans Med Imaging, 2001, 20(3): 233-239.
5.	Yu L, Chen H, Dou Q, et al. Automated melanoma recognition in dermoscopy images via very deep residual networks. IEEE Trans Med Imaging, 2017, 36(4): 994-1004.
6.	Codella N, Cai J, Abedini M, et al. Deep learning, sparse coding, and SVM for melanoma recognition in dermoscopy images// International Workshop on Machine Learning in Medical Imaging. Munich: MLMI, 2015: 118-126.
7.	Esteva A, Kuprel B, Novoa R A, et al. Dermatologist-levelclassification of skin cancer with deep neural networks. Nature, 2017, 542(7639): 115-118.
8.	Kawahara J, BenTaieb A, Hamarneh G. Deep features to classify skin lesions// 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). Prague: IEEE, 2016: 1397-1400.
9.	Ayas S. Multiclass skin lesion classification in dermoscopic images using swin transformer model. Neural Comput Appl, 2023, 35(9): 6713-6722.
10.	Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 10012-10022.
11.	Liu Z, Mao H, Wu C Y, et al. A convnet for the 2020s// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 11976-11986.
12.	Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module// Proceedings of the European Conference on Computer Vision (ECCV). Munich: ECCV, 2018: 3-19.
13.	Sandler M, Howard A, Zhu M, et al. Mobilenetv2: Inverted residuals and linear bottlenecks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 4510-4520.
14.	Romdhane T F, Alhichri H, Ouni R, et al. Electrocardiogram heartbeat classification based on a deep convolutional neural network and focal loss. Comput Biol Med, 2020, 123: 103866.
15.	Tschandl P, Rosendahl C, Kittler H. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci Data, 2018, 5(1): 180161.
16.	Combalia M, Codella N C F, Rotemberg V, et al. BCN20000: Dermoscopic lesions in the wild. arXiv, 2019, 2019: 1908.02288.
17.	Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 4700-4708.
18.	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE 2016: 770-778.
19.	Ali R, Hardie R C, Narayanan B N, et al. Deep learning ensemble methods for skin lesion analysis towards melanoma detection// 2019 IEEE National Aerospace and Electronics Conference (NAECON). Dayton: IEEE, 2019: 311-316.
20.	Pacheco A G C, Ali A R, Trappenberg T. Skin cancer detection based on deep learning and entropy to detect outlier samples. arXiv, 2019, 2019: 1909.04525.
21.	Ali R, Hardie R C, De Silva M S, et al. Skin lesion segmentation and classification for ISIC 2018 by combining deep CNN and handcrafted features. arXiv, 2019, 2019: 1908.05730.
22.	Ahmed S A A, Yanikoğlu B, Göksu Ö, et al. Skin lesion classification with deep CNN ensembles// 2020 28th Signal Processing and Communications Applications Conference (SIU). Gaziantep: IEEE, 2020: 1-4.
23.	Almaraz-Damian J A, Ponomaryov V, Sadovnychiy S, et al. Melanoma and nevus skin lesion classification using handcraft and deep learning feature fusion via mutual information measures. Entropy, 2020, 22(4): 484.
24.	Guissous A E. Skin lesion classification using deep neural network. arXiv, 2019, 2019: 1911.07817.
25.	Zhang J, Xie Y, Xia Y, et al. Attention residual learning for skin lesion classification. IEEE Trans Med Imaging, 2019, 38(9): 2092-2103.
26.	Benyahia S, Meftah B, Lézoray O. Multi-features extraction based on deep learning for skin lesion classification. Tissue Cell, 2022, 74: 101701.
27.	Afza F, Sharif M, Khan M A, et al. Multiclass skin lesion classification using hybrid deep features selection and extreme learning machine. Sensors, 2022, 22(3): 799.

1. Kumari S, Choudhary P K, Shukla R, et al. Recent advances in nanotechnology based combination drug therapy for skin cancer. J Biomater Sci Polym Ed, 2022, 33(11): 1435-1468.
2. Steiner A, Binder M, Schemper M, et al. Statistical evaluation of epiluminescence microscopy criteria for melanocytic pigmented skin lesions. J Am Acad Dermatol, 1993, 29(4): 581-588.
3. Celebi M E, Iyatomi H, Schaefer G, et al. Lesion border detection in dermoscopy images. Comput Med Imaging Graph, 2009, 33(2): 148-153.
4. Ganster H, Pinz P, Rohrer R, et al. Automated melanoma recognition. IEEE Trans Med Imaging, 2001, 20(3): 233-239.
5. Yu L, Chen H, Dou Q, et al. Automated melanoma recognition in dermoscopy images via very deep residual networks. IEEE Trans Med Imaging, 2017, 36(4): 994-1004.
6. Codella N, Cai J, Abedini M, et al. Deep learning, sparse coding, and SVM for melanoma recognition in dermoscopy images// International Workshop on Machine Learning in Medical Imaging. Munich: MLMI, 2015: 118-126.
7. Esteva A, Kuprel B, Novoa R A, et al. Dermatologist-levelclassification of skin cancer with deep neural networks. Nature, 2017, 542(7639): 115-118.
8. Kawahara J, BenTaieb A, Hamarneh G. Deep features to classify skin lesions// 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). Prague: IEEE, 2016: 1397-1400.
9. Ayas S. Multiclass skin lesion classification in dermoscopic images using swin transformer model. Neural Comput Appl, 2023, 35(9): 6713-6722.
10. Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 10012-10022.
11. Liu Z, Mao H, Wu C Y, et al. A convnet for the 2020s// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 11976-11986.
12. Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module// Proceedings of the European Conference on Computer Vision (ECCV). Munich: ECCV, 2018: 3-19.
13. Sandler M, Howard A, Zhu M, et al. Mobilenetv2: Inverted residuals and linear bottlenecks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 4510-4520.
14. Romdhane T F, Alhichri H, Ouni R, et al. Electrocardiogram heartbeat classification based on a deep convolutional neural network and focal loss. Comput Biol Med, 2020, 123: 103866.
15. Tschandl P, Rosendahl C, Kittler H. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci Data, 2018, 5(1): 180161.
16. Combalia M, Codella N C F, Rotemberg V, et al. BCN20000: Dermoscopic lesions in the wild. arXiv, 2019, 2019: 1908.02288.
17. Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 4700-4708.
18. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE 2016: 770-778.
19. Ali R, Hardie R C, Narayanan B N, et al. Deep learning ensemble methods for skin lesion analysis towards melanoma detection// 2019 IEEE National Aerospace and Electronics Conference (NAECON). Dayton: IEEE, 2019: 311-316.
20. Pacheco A G C, Ali A R, Trappenberg T. Skin cancer detection based on deep learning and entropy to detect outlier samples. arXiv, 2019, 2019: 1909.04525.
21. Ali R, Hardie R C, De Silva M S, et al. Skin lesion segmentation and classification for ISIC 2018 by combining deep CNN and handcrafted features. arXiv, 2019, 2019: 1908.05730.
22. Ahmed S A A, Yanikoğlu B, Göksu Ö, et al. Skin lesion classification with deep CNN ensembles// 2020 28th Signal Processing and Communications Applications Conference (SIU). Gaziantep: IEEE, 2020: 1-4.
23. Almaraz-Damian J A, Ponomaryov V, Sadovnychiy S, et al. Melanoma and nevus skin lesion classification using handcraft and deep learning feature fusion via mutual information measures. Entropy, 2020, 22(4): 484.
24. Guissous A E. Skin lesion classification using deep neural network. arXiv, 2019, 2019: 1911.07817.
25. Zhang J, Xie Y, Xia Y, et al. Attention residual learning for skin lesion classification. IEEE Trans Med Imaging, 2019, 38(9): 2092-2103.
26. Benyahia S, Meftah B, Lézoray O. Multi-features extraction based on deep learning for skin lesion classification. Tissue Cell, 2022, 74: 101701.
27. Afza F, Sharif M, Khan M A, et al. Multiclass skin lesion classification using hybrid deep features selection and extreme learning machine. Sensors, 2022, 22(3): 799.

Journal of Biomedical Engineering

Skin lesion classification with multi-level fusion of Swin-T and ConvNeXt

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content