Skip to main content
SpringerLink
Log in

Radiographs and texts fusion learning based deep networks for skeletal bone age assessment

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Bone age assessment is a pediatric examination that determines the difference between skeletal age and chronological age. The discrepancy between the two ages will often trigger the likelihood of genetic disorders, hormonal complications and abnormalities of maturity in the skeletal system. Recently, although some automated bone age assessment methods by analyzing radiographs have been researched, the available text data from radiological reports are not used. Texts and radiographs are two different modals, the fusion of them can give us much more information for bone age assessment. In this paper, we present a novel multi-modal data fusion-learning network, called RT-FuseNet, for bone age assessment utilizing radiographs and texts. Specifically, we develop a convolutional neural network with spatial pyramid pooling layer and attention mechanism module to ensure the integrity of the image space information and enhance the subtle difference of features among radiographs respectively. In addition, texts are incorporated into the learning model to jointly learn non-linear correlations between various heterogeneous data. To evaluate the proposed approach, two datasets are used and several neural network structures are compared. Experimental results show that the proposed approach performs well.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://www.kaggle.com/kmader/rsna-bone-age

  2. http://www.bonexpert.com

References

  1. Albanese A, Stanhope R (2010) Investigation of delayed puberty. Clin. Endocrinol. 43(1):105–110

    Article  Google Scholar 

  2. Athanasios C, Maria B, George K, Miranda A (2007) Bone age estimation and prediction of final height in patients with beta-thalassaemia major: A comparison between the two most common methods. In: Pediatric radiology, vol 37, pp 1241–1246

  3. Chen T, Ma X, Ying X, Wang W, Yuan C, Lu W, Chen D, Wu J (2019) Multi-modal fusion learning for cervical dysplasia diagnosis. In: IEEE 16th International Symposium on Biomedical Imaging (ISBI)

  4. Dai J, Qi H, Xiong Y, Li Y, Zhang G, Hu H, Wei Y (2017) Deformable convolutional networks. In: International Conference on Computer Vision

  5. Ferdinand H, Inga B, Heike R, Thomas D, Hauke S (2013) Bone age assessment using the classifying generalized hough transform. In: Pattern recognition, pp 313–322

  6. Gao F, Ge Y, Liu Y (2018) Remember and forget: Video and text fusion for video question answering. Multimed Tools Appl 77:29269–29282

    Article  Google Scholar 

  7. Greulich W, Pyle S (1959) Radiographic atlas of skeletal development of the hand and wrist. The American Journal of the Medical Sciences 238(3)

  8. Guo C, Chai Y, Wang C (2016) Multi-source heterogeneous data recognition based on linguistic labels. In: International conference on cyber-enabled distributed computing and knowledge discovery, pp. 278–285

  9. Hao P, Chokuwa S, Xie X, Wu F, Wu J, Bai C (2019) Skeletal bone age assessments for young children based on regression convolutional neural networks. Math. Biosci. Eng. 16(6):6454–6466

    Article  MathSciNet  Google Scholar 

  10. Hao P, You K, Feng H, Xu X, Zhang F, Wu F, Zhang P, Chen W (2019) Lung adenocarcinoma diagnosis in one stage. Neurocomputing

  11. He K, Zhang X, Ren S, Sun J (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37 (9):1904–1916

    Article  Google Scholar 

  12. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition

  13. Huang G, Liu Z, Maaten LVD, Weinberger KQ (2017) Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition

  14. Iglovikov V, Rakhlin A, Kalinin A, Shvets A (2018) Paediatric bone age assessment using deep convolutional neural networks. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp 300–308

  15. Lee H, Tajmir S, Lee J, Zissen M, Yeshiwas B, Alkasab T, Choy G, Do S (2017) Fully automated deep learning system for bone age assessment. J. Digit. Imaging 30(4):427–441

    Article  Google Scholar 

  16. Mutasa S, Chang PD, Ruzal-Shapiro C, Ayyala R (2018) Mabal: a novel deep-learning architecture for machine-assisted bone age labeling. J. Digit. Imaging 31:513–519

    Article  Google Scholar 

  17. Ren X, Li T, Yang X, Wang S, Ahmad S, Xiang L, Stone S, Li L, Zhan Y, Shen D, Wang Q (2018) Regression convolutional neural network for automated pediatric bone age assessment from hand radiograph. IEEE Journal of Biomedical and Health Informatics, pp 2168–2194

  18. Rijn RV, Thodberg HH (2013) Bone age assessment: automated techniques coming of age?. In: Acta Radiologica, vol 54, pp 1024–9

  19. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2019) Grad-cam: Visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis., pp 1–24

  20. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations

  21. Somkantha K, Nipon T, Auephanwiriyakul S (2011) Bone age assessment in young children using automatic carpal bone feature extraction and support vector regression. J. Digit. Imaging 24(6):1044–1058

    Article  Google Scholar 

  22. Souza D, Oliveira MM (2018) End-to-end bone age assessment with residual learning. In: 31st SIBGRAPI conference on graphics, patterns and images (SIBGRAPI), pp 197–203

  23. Spampinato C, Palazzo S, Giordamo D, Aldinucci M, Leonardi R (2017) Deep learning for automated skeletal bone age assessment in x-ray images. Med. Image Anal. 36:41–51

    Article  Google Scholar 

  24. Steenkiste TV, Ruyssinck J, Janssens OV (2018) et.al., V.F.V. Automated assessment of bone age using deep learning and gaussian process regression. In: 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 674–677

  25. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: IEEE Conference on computer vision and pattern recognition, pp 1–9

  26. Tanner J, Whitehouse R, Cameron N, Marshall W, Healy M, Goldstein H (1975) Assessment of skeletal maturity and prediction of adult height (TW2 method)

  27. Thodberg H, Kreiborg S, Juul A, Pedersen K (2009) The bonexpert method for automated determination of skeletal maturity. IEEE Trans. Med. Imaging 28(1):52–66

    Article  Google Scholar 

  28. Woo S, Park J, Lee J, Kweon IS (2018) Cbam: Convolutional block attention module. In: European Conference on Computer Vision (ECCV), pp 3–19

  29. Wu E, Kong B, Bai J, Lu Y, Gao F, Zhang S, et al. (2019) Residual attention based network for hand bone age assessment. In: IEEE 16th International Symposium on Biomedical Imaging (ISBI), pp 1158–1161

  30. Zhang L, Xie Y, Xidao L, Zhang X (2018) Multi-source heterogeneous data fusion. In: International conference on artificial intelligence and big data, pp 47–51

Download references

Acknowledgements

This project was supported by Zhejiang Provincial Natural Science Foundation of China (No. LY18F020034, LY18C130012), National Natural Science Foundation of China (No.61801428, 61672543), the Zhejiang University Education Foundation (No. K18-511120-004 and No. K17-518051-021), and the Major Scientific Project of Zhejiang Lab (No. 2018DG0ZX01).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Zhejiang University of Technology, Zhejiang, China

    Pengyi Hao, Taotao Ye, Xuhang Xie, Fuli Wu & Weilong Ding

  2. College of Educational Science and Technology, Zhejiang University of Technology, Zhejiang, China

    Wuheng Zuo

  3. The First Affiliated Hospital, Zhejiang University, Hangzhou, China

    Wei Chen

  4. State Key Lab of CAD & CG, Zhejiang University, Zhejiang, China

    Wei Chen

  5. College of Computer Science and Technology, Zhejiang University, Zhejiang, China

    Jian Wu

  6. Real Doctor Research Center, Zhejiang University, Zhejiang, China

    Jian Wu

  7. Institute of Artificial Intelligence, Guilin University of Electronic Technology, Guilin, China

    Xiaonan Luo

Authors
  1. Pengyi Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Taotao Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xuhang Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fuli Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weilong Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Wuheng Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Wei Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jian Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Xiaonan Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fuli Wu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hao, P., Ye, T., Xie, X. et al. Radiographs and texts fusion learning based deep networks for skeletal bone age assessment. Multimed Tools Appl 80, 16347–16366 (2021). https://doi.org/10.1007/s11042-020-08943-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08943-1

Keywords

Search

Navigation