Skip to main content
SpringerLink
Log in

Automated Pathogenesis-Based Diagnosis of Lumbar Neural Foraminal Stenosis via Deep Multiscale Multitask Learning

  • Original Article
  • Published:
Neuroinformatics Aims and scope Submit manuscript

Abstract

Pathogenesis-based diagnosis is a key step to prevent and control lumbar neural foraminal stenosis (LNFS). It conducts both early diagnosis and comprehensive assessment by drawing crucial pathological links between pathogenic factors and LNFS. Automated pathogenesis-based diagnosis would simultaneously localize and grade multiple spinal organs (neural foramina, vertebrae, intervertebral discs) to diagnose LNFS and discover pathogenic factors. The automated way facilitates planning optimal therapeutic schedules and relieving clinicians from laborious workloads. However, no successful work has been achieved yet due to its extreme challenges since 1) multiple targets: each lumbar spine has at least 17 target organs, 2) multiple scales: each type of target organ has structural complexity and various scales across subjects, and 3) multiple tasks, i.e., simultaneous localization and diagnosis of all lumbar organs, are extremely difficult than individual tasks. To address these huge challenges, we propose a deep multiscale multitask learning network (DMML-Net) integrating a multiscale multi-output learning and a multitask regression learning into a fully convolutional network. 1) DMML-Net merges semantic representations to reinforce the salience of numerous target organs. 2) DMML-Net extends multiscale convolutional layers as multiple output layers to boost the scale-invariance for various organs. 3) DMML-Net joins a multitask regression module and a multitask loss module to prompt the mutual benefit between tasks. Extensive experimental results demonstrate that DMML-Net achieves high performance (0.845 mean average precision) on T1/T2-weighted MRI scans from 200 subjects. This endows our method an efficient tool for clinical LNFS diagnosis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. Peripheral nervous system consists of the nerves and ganglia outside the brain and spinal cord, serving as a relay between the brain and spinal cord and the rest of the body.

  2. Pathogenesis of a disease is the biological mechanism (or mechanisms) that leads to the diseased state.

  3. https://github.com/balancap/SSD-Tensorflow.

  4. https://github.com/endernewton/tf-faster-rcnn.

References

  • Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro C., Corrado, G.S., Davis, A., Dean, J., Devin, M., et al (2016). Tensorflow: large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:160304467.

  • Alomari, R.S., Corso, J.J., Chaudhary, V. (2011). Labeling of lumbar discs using both pixel- and object-level features with a two-level probabilistic model. IEEE Transactions on Medical Imaging, 30(1), 1–10. https://doi.org/10.1109/TMI.2010.2047403.

    Article  PubMed  Google Scholar 

  • Ando, R.K. (2006). (2006). Applying alternating structure optimization to word sense disambiguation. In Proceedings of the tenth conference on computational natural language learning, association for computational linguistics (pp. 77–84).

  • Ando, RK, & Zhang, T. (2005). A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6, 1817–1853.

    Google Scholar 

  • Baxter, J. et al. (2000). A model of inductive bias learning. Journal of Artificial Intelligence Research (JAIR), 12(149–198), 3.

    Google Scholar 

  • Bell, S., Lawrence Zitnick, C., Bala, K., Girshick, R. (2016). Inside-outside net: detecting objects in context with skip pooling and recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2874–2883).

  • Ben-David, S, & Schuller, R. (2003). Exploiting task relatedness for multiple task learning. In Learning theory and kernel machines (pp. 567–580). Springer.

  • Cai, Y., Osman, S., Sharma, M., Landis, M., Li, S. (2015). Multi-modality vertebra recognition in arbitrary views using 3d deformable hierarchical model. IEEE Transactions on Medical Imaging, 34(8), 1676–1693. https://doi.org/10.1109/TMI.2015.2392054 https://doi.org/10.1109/TMI.2015.2392054.

    Article  PubMed  Google Scholar 

  • Cai, Y., Leungb, S., Warringtonb, J., Pandeyb, S., Shmuilovichb, O., Lib, S. (2017). Direct spondylolisthesis identification and measurement in mr/ct using detectors trained by articulated parameterized spine model. In Proc. of SPIE (Vol. 10133, pp. 1013,319–1).

  • Chen, X, & Gupta, A. (2017). An implementation of faster rcnn with study for region sampling. arXiv preprint arXiv:http://arXiv.org/abs/170202138.

  • Cinotti, G., De Santis, P., Nofroni, I., Postacchini, F. (2002). Stenosis of lumbar intervertebral foramen: anatomic study on predisposing factors. Spine, 27(3), 223–229.

    Article  PubMed  Google Scholar 

  • Corso, J.J., Raja’S, A., Chaudhary, V. (2008). Lumbar disc localization and labeling with a probabilistic model on both pixel and object features. In International conference on medical image computing and computer-assisted intervention (pp. 202–210). Springer.

  • Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A. (2010). The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2), 303–338.

    Article  Google Scholar 

  • Evgeniou, T., Pontil, M., Toubia, O. (2007). A convex optimization approach to modeling consumer heterogeneity in conjoint estimation. Marketing Science, 26(6), 805–818.

    Article  Google Scholar 

  • Ghosha, S., Raja’S, A., Chaudharya, V., Dhillonb, G. (2011). Automatic lumbar vertebra segmentation from clinical ct for wedge compression fracture diagnosis. Work, 9, 11.

    Google Scholar 

  • Girshick, R. (2015). Fast r-cnn. In Proceedings of the IEEE international conference on computer vision (pp. 1440–1448).

  • Hariharan, B, Arbeláez, P, Girshick, R, Malik, J. (2015). Hypercolumns for object segmentation and fine-grained localization. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 447–456).

  • Hasegawa, T., An, H.S., Haughton, V.M., Nowicki, B.H. (1995). Lumbar foraminal stenosis: critical heights of the intervertebral discs and foramina. a cryomicrotome study in cadavera. Journal of Bone and Joint Surgery (American), 77(1), 32–38.

    Article  CAS  Google Scholar 

  • He, X., Yin, Y., Sharma, M., Brahm, G., Mercado, A., Li, S. (2016). Automated diagnosis of neural foraminal stenosis using synchronized superpixels representation. In MICCAI (2) (pp. 335–343). Springer.

  • He, X., Landisa, M., Leunga, S., Warringtona, J., Shmuilovicha, O., Lia, S. (2017a). Automated grading of lumbar disc degeneration via supervised distance metric learning. In Proc. of SPIE Vol (Vol. 10134, pp. 1013,443-1).

  • He, X., Lum, A., Sharma, M., Brahm, G., Mercado, A., Li, S. (2017b). Automated segmentation and area estimation of neural foramina with boundary regression model. Pattern Recognition, 63, 625–641.

  • He, X., Zhang, H., Landis, M., Sharma, M., Warrington, J., Li, S. (2017c). Unsupervised boundary delineation of spinal neural foramina using a multi-feature and adaptive spectral segmentation. Medical Image Analysis, 36, 22–40.

  • Hoang Ngan Le, T., Zheng, Y., Zhu, C., Luu, K., Savvides, M. (2016). Multiple scale faster-rcnn approach to driver’s cell-phone usage and hands on steering wheel detection. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 46–53).

  • Hoiem, D., Chodpathumwan, Y., Dai, Q. (2012). Diagnosing error in object detectors. Computer Vision–ECCV, 2012, 340–353.

    Google Scholar 

  • Huang, S.H., Chu, Y.H., Lai, S.H., Novak, C.L. (2009). Learning-based vertebra detection and iterative normalized-cut segmentation for spinal mri. IEEE Transactions on Medical Imaging, 28(10), 1595–1605.

    Article  PubMed  Google Scholar 

  • Jamaludin, A., Kadir, T., Zisserman, A. (2017). Spinenet: automated classification and evidence visualization in spinal mris. Medical Image Analysis, 41, 63–73. https://doi.org/10.1016/j.media.2017.07.002 https://doi.org/10.1016/j.media.2017.07.002. http://www.sciencedirect.com/science/article/pii/S136184151730110X, special Issue on the 2016 Conference on Medical Image Computing and Computer Assisted Intervention (Analog to MICCAI 2015).

    Article  PubMed  Google Scholar 

  • Kaneko, Y., Matsumoto, M., Takaishi, H., Nishiwaki, Y., Momoshima, S., Toyama, Y. (2012). Morphometric analysis of the lumbar intervertebral foramen in patients with degenerative lumbar scoliosis by multidetector-row computed tomography. European Spine Journal, 21(12), 2594–2602.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kelm, B.M., Wels, M., Zhou, S.K., Seifert, S., Suehling, M., Zheng, Y., Comaniciu, D. (2013). Spine detection in ct and mr using iterated marginal space learning. Medical Image Analysis, 17(8), 1283–1292.

    Article  Google Scholar 

  • Kim, S., Lee, J.W., Chai, J.W., Yoo, H.J., Kang, Y., Seo, J., Ahn, J.M., Kang, H.S. (2015). A new mri grading system for cervical foraminal stenosis based on axial t2-weighted images. Korean Journal of Radiology, 16(6), 1294–1302.

    Article  PubMed  PubMed Central  Google Scholar 

  • Klinder, T., Wolz, R., Lorenz, C., Franz, A., Ostermann, J. (2008). Spine segmentation using articulated shape models. Medical Image Computing and Computer-Assisted Intervention–MICCAI, 2008, 227–234.

    Google Scholar 

  • Law, M.W., Tay, K., Leung, A., Garvin, G.J., Li, S. (2013a). Intervertebral disc segmentation in mr images using anisotropic oriented flux. Medical Image Analysis, 17(1), 43–61. https://doi.org/10.1016/j.media.2012.06.006. http://www.sciencedirect.com/science/article/pii/S1361841512000874.

  • Law, M.W.K., Garvin, G.J., Tummala, S., Tay, K., Leung, A.E., Li, S. (2013b). Gradient competition anisotropy for centerline extraction and segmentation of spinal cords (pp. 49–61). Berlin: Springer.

  • Lee, S., Lee, J.W., Yeom, J.S., Kim, K.J., Kim, H.J., Chung, S.K., Kang, H.S. (2010). A practical mri grading system for lumbar foraminal stenosis. American Journal of Roentgenology, 194(4), 1095–1098.

    Article  PubMed  Google Scholar 

  • Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C. (2016). Ssd: single shot multibox detector. In European conference on computer vision (pp. 21–37). Springer.

  • Luo, W., Li, Y., Urtasun, R., Zemel, R. (2016). Understanding the effective receptive field in deep convolutional neural networks. In Advances in neural information processing systems (pp. 4898–4906).

  • Panjabi, M.M., Maak, T.G., Ivancic, P.C., Ito, S. (2006). Dynamic intervertebral foramen narrowing during simulated rear impact. Spine, 31(5), E128–E134.

    Article  PubMed  Google Scholar 

  • Park, H.J., Kim, S., Lee, S.Y., Park, N.H., Rho, M.H., Hong, H.P., Kwag, H.J., Kook, S.H., Choi, S.H. (2012). Clinical correlation of a new mr imaging method for assessing lumbar foraminal stenosis. American Journal of Neuroradiology, 33(5), 818–822.

    Article  PubMed  CAS  Google Scholar 

  • Peng, Z., Zhong, J., Wee, W., Lee, J.H. (2006). Automated vertebra detection and segmentation from the whole spine mr images. In 27th annual international conference of the engineering in medicine and biology society, 2005. IEEE-EMBS 2005 (pp. 2527–2530). IEEE.

  • Rajaee, S.S., Bae, H.W., Kanim, L.E., Delamarter, R.B. (2012). Spinal fusion in the united states: analysis of trends from 1998 to 2008. Spine, 37(1), 67–76.

    Article  PubMed  Google Scholar 

  • Raja’S, A., Corso, J.J., Chaudhary, V., Dhillon, G. (2011). Toward a clinical lumbar cad: herniation diagnosis. International Journal of Computer Assisted Radiology and Surgery, 6(1), 119–126.

    Article  Google Scholar 

  • Ren, S., He, K., Girshick, R., Sun, J. (2015). Faster r-cnn: towards real-time object detection with region proposal networks. In Advances in neural information processing systems (pp. 91–99).

  • Shi, R., Sun, D., Qiu, Z., Weiss, K.L. (2007). An efficient method for segmentation of mri spine images. In IEEE/ICME international conference on complex medical engineering, 2007. CME 2007 (pp. 713–717). IEEE.

  • Simonyan, K, Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:http://arXiv.org/abs/14091556.

  • Ṡtern, D, Likar, B, Pernuṡ, F, Vrtovec, T. (2009). Automated detection of spinal centrelines, vertebral bodies and intervertebral discs in ct and mr images of lumbar spine. Physics in Medicine and Biology, 55(1), 247.

    Article  Google Scholar 

  • Sun, X., Wu, P., Hoi, S.C. (2017). Face detection using deep learning: an improved faster rcnn approach. arXiv preprint arXiv:http://arXiv.org/abs/170108289.

  • Torralba, A., Murphy, K.P., Freeman, W.T. (2004). Sharing features: efficient boosting procedures for multiclass object detection. In Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004 (Vol. 2, pp. II–762–II–769). https://doi.org/10.1109/CVPR.2004.1315241 https://doi.org/10.1109/CVPR.2004.1315241.

  • Wan, S., Chen, Z., Zhang, T., Zhang, B., Wong, K.K. (2016). Bootstrapping face detection with hard negative examples. arXiv preprint arXiv:http://arXiv.org/abs/160802236.

  • Wang, Q., Lu, L., Wu, D., El-Zehiry, N., Zheng, Y., Shen, D., Zhou, K.S. (2015a). Automatic segmentation of spinal canals in ct images via iterative topology refinement. IEEE Transactions on Medical Imaging, 34(8), 1694–1704. https://doi.org/10.1109/TMI.2015.2436693.

  • Wang, Z, Zhen, X, Tay, K, Osman, S, Romano, W, Li, S. (2015b). Regression segmentation for m3 spinal images. IEEE Transactions on Medical Imaging, 34(8), 1640–1648.

  • Yan, C., Zhang, Y., Xu, J., Dai, F., Li, L., Dai, Q., Wu, F. (2014a). A highly parallel framework for hevc coding unit partitioning tree decision on many-core processors. IEEE Signal Processing Letters, 21(5), 573–576. https://doi.org/10.1109/LSP.2014.2310494.

  • Yan, C., Zhang, Y., Xu, J., Dai, F., Zhang, J., Dai, Q., Wu, F. (2014b). Efficient parallel framework for hevc motion estimation on many-core processors. IEEE Transactions on Circuits and Systems for Video Technology, 24(12), 2077–2089. https://doi.org/10.1109/TCSVT.2014.2335852.

  • Yan, C., Xie, H., Liu, S., Yin, J., Zhang, Y., Dai, Q. (2017a). Effective uyghur language text detection in complex background images for traffic prompt identification. IEEE Transactions on Intelligent Transportation Systems, PP(99), 1–10. https://doi.org/10.1109/TITS.2017.2749977.

  • Yan, C., Xie, H., Yang, D., Yin, J., Zhang, Y., Dai, Q. (2017b). Supervised hash coding with deep neural network for environment perception of intelligent vehicles. IEEE Transactions on Intelligent Transportation Systems, PP(99), 1–12. https://doi.org/10.1109/TITS.2017.2749965.

  • Yao, J., Burns, J.E., Forsberg, D., Seitel, A., Rasoulian, A., Abolmaesumi, P., Hammernik, K., Urschler, M., Ibragimov, B., Korez, R., et al (2016). A multi-center milestone study of clinical vertebral ct segmentation. Computerized Medical Imaging and Graphics, 49, 16–28.

    Article  PubMed  PubMed Central  Google Scholar 

  • Zhan, Y., Maneesh, D., Harder, M., Zhou, X.S. (2012). Robust mr spine detection using hierarchical learning and local articulated model. In International conference on medical image computing and computer-assisted intervention (pp. 141–148). Springer.

Download references

Acknowledgements

This work was made possible through support from Natural Science Foundation of Shandong Province in China (ZR2015FM010), Project of Shandong Province Higher Educational Science and Technology Program in China (No. J15LN20), Project of Shandong Province Traditional Chinese Medicine Technology Development Program in China (2015-026, 2017-001), and Project of Shandong Province Medical and Health Technology Development Program in China (No. 2016WS0577).

Author information

Authors and Affiliations

  1. College of Science and Technology, Shandong University of Traditional Chinese Medicine, Jinan, 250355, China

    Zhongyi Han & Benzheng Wei

  2. Computational Medicine Lab, Shandong University of Traditional Chinese Medicine, Jinan, 250355, China

    Zhongyi Han & Benzheng Wei

  3. Department of Medical Imaging, Western University, London, N6A 4V2, Canada

    Zhongyi Han & Shuo Li

  4. Digital Image Group (DIG), London, ON, N6A 4V2, Canada

    Zhongyi Han, Stephanie Leung, Ilanit Ben Nachum, David Laidley & Shuo Li

Authors
  1. Zhongyi Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Benzheng Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stephanie Leung
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ilanit Ben Nachum
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David Laidley
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shuo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Benzheng Wei or Shuo Li.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, Z., Wei, B., Leung, S. et al. Automated Pathogenesis-Based Diagnosis of Lumbar Neural Foraminal Stenosis via Deep Multiscale Multitask Learning. Neuroinform 16, 325–337 (2018). https://doi.org/10.1007/s12021-018-9365-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12021-018-9365-1

Keywords

Search

Navigation