Cross-modal retrieval method for thyroid ultrasound image and text based on generative adversarial network_Journal of Biomedical Engineering

Authors：

XU Feng ^1,3 , MA Xiaoping ² ,  LIU Libo ¹

1. School of Information Engineering, Ningxia University, Yinchuan 750021, P.R China;
2. Department of Medical Technology, The First People’s Hospital of Yinchuan, Yinchuan 750002, P.R. China;
3. Key Laboratory for Early Warning and Risk Management of Agricultural Meteorological Disasters In Dry Zones of China Meteorological Administration, Yinchuan 750006, P.R. China;

Corresponding author：

LIU Libo, Email: liulib@163.com

Keywords：

thyroid; cross-modal retrieval; generative adversarial network; deep learning

DOI：

10.7507/1001-5515.201812042

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Ultrasonic examination is a common method in thyroid examination, and the results are mainly composed of thyroid ultrasound images and text reports. Implementation of cross modal retrieval method of images and text reports can provide great convenience for doctors and patients, but currently there is no retrieval method to correlate thyroid ultrasound images with text reports. This paper proposes a cross-modal method based on the deep learning and improved cross-modal generative adversarial network: ①the weight sharing constraints between the fully connection layers used to construct the public representation space in the original network are changed to cosine similarity constraints, so that the network can better learn the common representation of different modal data; ②the fully connection layer is added before the cross-modal discriminator to merge the full connection layer of image and text in the original network with weight sharing. Semantic regularization is realized on the basis of inheriting the advantages of the original network weight sharing. The experimental results show that the mean average precision of cross modal retrieval method for thyroid ultrasound image and text report in this paper can reach 0.508, which is significantly higher than the traditional cross-modal method, providing a new method for cross-modal retrieval of thyroid ultrasound image and text report.

Citation： XU Feng, MA Xiaoping, LIU Libo. Cross-modal retrieval method for thyroid ultrasound image and text based on generative adversarial network. Journal of Biomedical Engineering, 2020, 37(4): 641-651. doi: 10.7507/1001-5515.201812042 Copy

1.	韩婧, 康骅. 甲状腺癌的发病现状及影响因素. 实用预防医学, 2018, 25(7): 894-897.
2.	Hotelling H. Relations between two sets of variates. Biometrika, 1936, 28(3/4): 321-377.
3.	Hardoon D R, Szedmak S, Shawe-Taylor J. Canonical correlation analysis: an overview with application to learning methods. Neural Comput, 2004, 16(12): 2639-2664.
4.	Rasiwasia N, Pereira J C, Coviello E, et al. A new approach to cross-modal multimedia retrieval//Proceedings of the 18th ACM International Conference on Multimedia. New York: ACM Press, 2010: 251-260.
5.	Gong Yunchao, Ke Qifa, Michael I, <italic>et al</italic>. A multi-view embedding space for modeling internet images. Int J Comput Vis, 2012, 106(2): 210-233.
6.	Li Dongge, Nevenka D, Li Mingkun, et al. Multimedia content processing through cross-modal association//Proceedings of the 11th ACM International Conference on Multimedia. New York: ACM Press, 2003: 604-611.
7.	Zhai Xiaohua, Peng Yuxin, Xiao Jianguo. Heterogeneous metric learning with joint graph regularization for cross-media retrieval//Proc of the 27th AAAI Conference on Artificial Intelligence, Washington: AAAI Press, 2013: 1198-1204.
8.	Zhai Xiaohua, Peng Yuxin, Xiao Jianguo. Learning cross-media joint representation with sparse and semi-supervised regularization. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(6): 965-978.
9.	Wang Kaiye, He Ran, Wang Liang, <italic>et al</italic>. Joint feature selection and subspace learning for cross-modal retrieval. IEEE Trans Pattern Anal Mach Intell, 2016, 38(10): 2010-2023.
10.	Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60(6): 84-90.
11.	Ren Shaoqing, He Kaiming, Girshick R, <italic>et al</italic>. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell, 2017, 39(6): 1137-1149.
12.	Jiquan N, Aditya K, Mingyu K, et al. Multimodal deep learning//Proc of the 28th International Conference on Machine Learning, Bellevue: ICML Press, 2011: 689-696.
13.	Feng Fangxiang, Wang Xiaojie, Li Ruifan. Cross-modal retrieval with correspondence autoencoder//Proceedings of the ACM International Conference on Multimedia - MM '14, New York: ACM Press, 2014: 7-16.
14.	Galen A, Raman A, Jeff B, et al. Deep canonical correlation analysis//Proc of the 30th International Conference on Mache Learning, Atlanta: ACM Press, 2013: 1247-1255.
15.	Fei Yan, Mikolajczyk K. Deep correlation for matching images and text//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston: IEEE, 2015: 3441-3450.
16.	Peng Yuxin, Qi Jinwei, Xin Huang, <italic>et al</italic>. CCL: cross-modal correlation learning with multigrained fusion by hierarchical network. IEEE Transactions on Multimedia, 2018, 20(2): 405-420.
17.	Ian J G, Jean P A, Mehdi M, et al. Generative adversarial nets//Proc of the 27th International Conference on Neural Information Processing Systems, Cambridge: MIT Press, 2014: 2672-2680.
18.	Augustus O, Christopher O, Jonathon S. Conditional image synthesis with auxiliary classifier GANs//Proceedings of the 34th International Conference on Machine Learning, Sydney: ACM Press, 2017: 2642-2651.
19.	Scott R, Zeynep A, Santosh M, et al. Learning what and where to draw//Proc of the 30th International Conference on Neural Information Processing Systems. Barcelona: ACM Press, 2016: 217–225.
20.	Zhang Han, Xu Tao, Li Hongsheng, et al. StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks//2017 IEEE International Conference on Computer Vision (ICCV), Venice: IEEE. 2017: 1-8.
21.	Peng Yuxin, Qi Jinwei. CM-GANs: cross-modal generative adversarial networks for common representation learning. ACM Transactions on Multimedia Computing, Communications, and Applications, 2019, 15(1): 22.
22.	Karen S, Andrew Z. Very deep convolutional networks for large-scale image recognition. (2014-9-4). [2015-4-25]. https://arxiv.org/abs/1409.1556.
23.	Tomas M, Ilya S, Kai C, et al. Distributed representations of words and phrases and their compositionality// Proc of the 26th International Conference on Neural Information Processing Systems, Lake Tahoe: ACM Press, 2013: 3111–3119.

1. 韩婧, 康骅. 甲状腺癌的发病现状及影响因素. 实用预防医学, 2018, 25(7): 894-897.
2. Hotelling H. Relations between two sets of variates. Biometrika, 1936, 28(3/4): 321-377.
3. Hardoon D R, Szedmak S, Shawe-Taylor J. Canonical correlation analysis: an overview with application to learning methods. Neural Comput, 2004, 16(12): 2639-2664.
4. Rasiwasia N, Pereira J C, Coviello E, et al. A new approach to cross-modal multimedia retrieval//Proceedings of the 18th ACM International Conference on Multimedia. New York: ACM Press, 2010: 251-260.
5. Gong Yunchao, Ke Qifa, Michael I, <italic>et al</italic>. A multi-view embedding space for modeling internet images. Int J Comput Vis, 2012, 106(2): 210-233.
6. Li Dongge, Nevenka D, Li Mingkun, et al. Multimedia content processing through cross-modal association//Proceedings of the 11th ACM International Conference on Multimedia. New York: ACM Press, 2003: 604-611.
7. Zhai Xiaohua, Peng Yuxin, Xiao Jianguo. Heterogeneous metric learning with joint graph regularization for cross-media retrieval//Proc of the 27th AAAI Conference on Artificial Intelligence, Washington: AAAI Press, 2013: 1198-1204.
8. Zhai Xiaohua, Peng Yuxin, Xiao Jianguo. Learning cross-media joint representation with sparse and semi-supervised regularization. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(6): 965-978.
9. Wang Kaiye, He Ran, Wang Liang, <italic>et al</italic>. Joint feature selection and subspace learning for cross-modal retrieval. IEEE Trans Pattern Anal Mach Intell, 2016, 38(10): 2010-2023.
10. Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60(6): 84-90.
11. Ren Shaoqing, He Kaiming, Girshick R, <italic>et al</italic>. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell, 2017, 39(6): 1137-1149.
12. Jiquan N, Aditya K, Mingyu K, et al. Multimodal deep learning//Proc of the 28th International Conference on Machine Learning, Bellevue: ICML Press, 2011: 689-696.
13. Feng Fangxiang, Wang Xiaojie, Li Ruifan. Cross-modal retrieval with correspondence autoencoder//Proceedings of the ACM International Conference on Multimedia - MM '14, New York: ACM Press, 2014: 7-16.
14. Galen A, Raman A, Jeff B, et al. Deep canonical correlation analysis//Proc of the 30th International Conference on Mache Learning, Atlanta: ACM Press, 2013: 1247-1255.
15. Fei Yan, Mikolajczyk K. Deep correlation for matching images and text//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston: IEEE, 2015: 3441-3450.
16. Peng Yuxin, Qi Jinwei, Xin Huang, <italic>et al</italic>. CCL: cross-modal correlation learning with multigrained fusion by hierarchical network. IEEE Transactions on Multimedia, 2018, 20(2): 405-420.
17. Ian J G, Jean P A, Mehdi M, et al. Generative adversarial nets//Proc of the 27th International Conference on Neural Information Processing Systems, Cambridge: MIT Press, 2014: 2672-2680.
18. Augustus O, Christopher O, Jonathon S. Conditional image synthesis with auxiliary classifier GANs//Proceedings of the 34th International Conference on Machine Learning, Sydney: ACM Press, 2017: 2642-2651.
19. Scott R, Zeynep A, Santosh M, et al. Learning what and where to draw//Proc of the 30th International Conference on Neural Information Processing Systems. Barcelona: ACM Press, 2016: 217–225.
20. Zhang Han, Xu Tao, Li Hongsheng, et al. StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks//2017 IEEE International Conference on Computer Vision (ICCV), Venice: IEEE. 2017: 1-8.
21. Peng Yuxin, Qi Jinwei. CM-GANs: cross-modal generative adversarial networks for common representation learning. ACM Transactions on Multimedia Computing, Communications, and Applications, 2019, 15(1): 22.
22. Karen S, Andrew Z. Very deep convolutional networks for large-scale image recognition. (2014-9-4). [2015-4-25]. https://arxiv.org/abs/1409.1556.
23. Tomas M, Ilya S, Kai C, et al. Distributed representations of words and phrases and their compositionality// Proc of the 26th International Conference on Neural Information Processing Systems, Lake Tahoe: ACM Press, 2013: 3111–3119.

Journal of Biomedical Engineering

Cross-modal retrieval method for thyroid ultrasound image and text based on generative adversarial network

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content