Skip to main content
Springer Nature Link
Log in

Large-scale Exploration of Neuronal Morphologies Using Deep Learning and Augmented Reality

  • Original Article
  • Published:
Neuroinformatics Aims and scope Submit manuscript

Abstract

Recently released large-scale neuron morphological data has greatly facilitated the research in neuroinformatics. However, the sheer volume and complexity of these data pose significant challenges for efficient and accurate neuron exploration. In this paper, we propose an effective retrieval framework to address these problems, based on frontier techniques of deep learning and binary coding. For the first time, we develop a deep learning based feature representation method for the neuron morphological data, where the 3D neurons are first projected into binary images and then learned features using an unsupervised deep neural network, i.e., stacked convolutional autoencoders (SCAEs). The deep features are subsequently fused with the hand-crafted features for more accurate representation. Considering the exhaustive search is usually very time-consuming in large-scale databases, we employ a novel binary coding method to compress feature vectors into short binary codes. Our framework is validated on a public data set including 58,000 neurons, showing promising retrieval precision and efficiency compared with state-of-the-art methods. In addition, we develop a novel neuron visualization program based on the techniques of augmented reality (AR), which can help users take a deep exploration of neuron morphologies in an interactive and immersive manner.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of deep networks. NIPS, 19, 153.

    Google Scholar 

  • Cannon, R.C., Turner, D.A., Pyapali, G.K., & Wheal, H.V. (1998). An on-line archive of reconstructed hippocampal neurons. Journal of Neuroscience Methods, 84(1), 49–54.

    Article  PubMed  CAS  Google Scholar 

  • Conjeti, S., Katouzian, A., Kazi, A., Mesbah, S., Beymer, D., Syeda-Mahmood, T.F., & Navab, N. (2016a). Metric hashing forests. Medical image analysis, 34, 13–29.

  • Conjeti, S., Mesbah, S., Negahdar, M., Rautenberg, P.L., Zhang, S., Navab, N., & Katouzian, A. (2016b). Neuron-miner: an advanced tool for morphological search and retrieval in neuroscientific image databases. Neuroinformatics, 14(4), 369–385.

  • Costa, L.D.F., Zawadzki, K., Miazaki, M., Viana, M.P., & Taraskin, S. (2010). Unveiling the neuromorphological space. Frontiers in Computational Neuroscience, 4, 150–163.

    Article  PubMed Central  Google Scholar 

  • Costa, M., Manton, J.D., Ostrovsky, A.D., Prohaska, S., & Jefferis, G.S. (2016). NBLAST: Rapid, sensitive comparison of neuronal structure and construction of neuron family databases. Neuron, 91(2), 293–311.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Gong, Y., Lazebnik, S., Gordo, A., & Perronnin, F. (2013). Iterative quantization: a procrustean approach to learning binary codes for large-scale image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12), 2916–2929.

    Article  PubMed  Google Scholar 

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR) (pp. 770–778).

  • Hinton, G.E., & Salakhutdinov, R.R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507.

    Article  PubMed  CAS  Google Scholar 

  • Ioffe, S., & Szegedy, C. (2015). Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML) (pp. 448–456).

  • Jain, A., Nandakumar, K., & Ross, A. (2005). Score normalization in multimodal biometric systems. Pattern recognition, 38(12), 2270–2285.

    Article  Google Scholar 

  • Ji, S. (2013). Computational genetic neuroanatomy of the developing mouse brain: dimensionality reduction, visualization, and clustering. BMC bioinformatics, 14(1), 222.

    Article  PubMed  PubMed Central  Google Scholar 

  • Krizhevsky, A., Sutskever, I., & Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS) (pp. 1097–1105).

  • LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444.

    Article  PubMed  CAS  Google Scholar 

  • LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324.

    Article  Google Scholar 

  • Li, R., Zeng, T., Peng, H., & Ji, S. (2017a). Deep learning segmentation of optical microscopy images improves 3-D neuron reconstruction. IEEE Transactions on Medical Imaging, 36(7), 1533–1541.

  • Li, Z., Fang, R., Shen, F., Katouzian, A., & Zhang, S. (2017b). Indexing and mining large-scale neuron databases using maximum inner product search. Pattern Recognition, 63, 680–688.

  • Li, Z., Metaxas, D.N., Lu, A., & Zhang, S. (2017c). Interactive exploration for continuously expanding neuron databases. Methods, 115, 100–109.

  • Li, Z., Shen, F., Fang, R., Conjeti, S., Katouzian, A., & Zhang, S. (2016). Maximum inner product search for morphological retrieval of large-scale neuron data.. In International Symposium on Biomedical Imaging (ISBI) (pp. 602–606).

  • Li, Z., Zhang, X., Mller, H., & Zhang, S. (2018). Large-scale retrieval for medical image analytics: A comprehensive review. Medical Image Analysis, 43, 66–84.

    Article  PubMed  Google Scholar 

  • Liu, J., Zhang, S., Liu, W., Deng, C., Zheng, Y., & Metaxas, D.N. (2017). Scalable mammogram retrieval using composite anchor graph hashing with iterative quantization. IEEE Transactions on Circuits and Systems for Video Technology, 27(11), 2450–2460.

    Article  Google Scholar 

  • Liu, W., Wang, J., Ji, R., Jiang, Y.G., & Chang, S.F. (2012). Super vised hashing with kernels. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2074–2081).

  • Liu, W., Wang, J., Kumar, S., & Chang, S.F. (2011). Hashing with graphs. In International Conference on Machine Learning (ICML) (pp. 1–8).

  • Masci, J., Meier, U., Ciresan, D., & Schmidhuber, J. (2011). Stacked convolutional auto-encoders for hierarchical feature extraction. ICANN, 52–59.

  • Mesbah, S., Conjeti, S., Kumaraswamy, A., Rautenberg, P., Navab, N., & Katouzian, A. (2015). Hashing forests for morphological search and retrieval in neuroscientific image databases. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) (pp. 52–59).

  • Mukherjee, S., Basu, S., Condron, B., & Acton, S.T. (2013). Tree2Tree2: neuron tracing in 3D. In International Symposium on Biomedical Imaging (ISBI) (pp. 448–451).

  • Nair, V., & Hinton, G.E. (2010). Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning (ICML) (pp. 807–814).

  • Peng, H., Ruan, Z., Long, F., Simpson, J.H., & Myers, E.W. (2010). V3D enables real-time 3D visualization and quantitative analysis of large-scale biological image data sets. Nature biotechnology, 28(4), 348–353.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Salakhutdinov, R. (2015). Learning deep generative models. Annual Review of Statistics and Its Application, 2, 361–385.

    Article  Google Scholar 

  • Scorcioni, R., Polavaram, S., & Ascoli, G.A. (2008). L-Measure: a web-accessible tool for the analysis, comparison and search of digital reconstructions of neuronal morphologies. Nature protocols, 3(5), 866–876.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Shen, F., Liu, W., Zhang, S., Yang, Y., & Shen, H.T. (2015). Learning binary codes for maximum inner product search. In IEEE International Conference on Computer Vision (ICCV) (pp. 4148–4156).

  • Shen, F., Yang, Y., Liu, L., Liu, W., Tao, D., & Shen, H.T. (2017). Asymmetric binary coding for image search. IEEE Transactions on Multimedia, 19(9), 2022–2032.

    Article  Google Scholar 

  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition, arXiv:1409.1556.

  • Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1–9).

  • Wan, Y., Long, F., Qu, L., Xiao, H., Hawrylycz, M., Myers, E.W., & Peng, H. (2015). BlastNeuron for automated comparison, retrieval and clustering of 3D neuron morphologies. Neuroinformatics, 13(4), 487–499.

    Article  PubMed  Google Scholar 

  • Wang, J., Liu, W., Kumar, S., & Chang, S.F. (2016). Learning to hash for indexing big dataa survey. Proceedings of the IEEE, 104(1), 34–57.

    Article  Google Scholar 

  • Weiss, Y., Torralba, A., & Fergus, R. (2009). Spectral hashing. In Advances in Neural Information Processing Systems (NIPS) (pp. 1753–1760).

  • Wu, G., Jia, H., Wang, Q., & Shen, D. (2011). SharpMean: groupwise registration guided by sharp mean image and tree-based registration. NeuroImage, 56(4), 1968–1981.

    Article  PubMed  PubMed Central  Google Scholar 

  • Yan, C., Zhang, Y., Dai, F., Wang, X., Li, L., & Dai, Q. (2014a). Parallel deblocking filter for HEVC on many-core processor. Electronics Letters, 50(5), 367–368.

  • Yan, C., Zhang, Y., Xu, J., Dai, F., Li, L., Dai, Q., & Wu, F. (2014b). A highly parallel framework for HEVC coding unit partitioning tree decision on many-core processors. IEEE Signal Processing Letters, 21(5), 573–576.

  • Yan, C., Zhang, Y., Xu, J., Dai, F., Zhang, J., Dai, Q., & Wu, F. (2014c). Efficient parallel framework for HEVC motion estimation on many-core processors. IEEE Transactions on Circuits and Systems for Video Technology, 24(12), 2077–2089.

  • Yu, G., & Yuan, J. (2014). Scalable forest hashing for fast similarity search. In IEEE International Conference on Multimedia and Expo (ICME) (pp. 1–6).

  • Zeiler, M.D., Taylor, G.W., & Fergus, R. (2011). Adaptive deconvolutional networks for mid and high level feature learning. In IEEE International Conference on Computer Vision (ICCV) (pp. 2018–2025).

  • Zhang, S., Yang, M., Cour, T., Yu, K., & Metaxas, D.N. (2015a). Query specific rank fusion for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(4), 803–815.

  • Zhang, X., Dou, H., Ju, T., Xu, J., & Zhang, S. (2016). Fusing heterogeneous features from stacked sparse autoencoder for histopathological image analysis. IEEE journal of biomedical and health informatics, 20(5), 1377–1383.

    Article  PubMed  Google Scholar 

  • Zhang, X., Liu, W., Dundar, M., Badve, S., & Zhang, S. (2015b). Towards large-scale histopathological image analysis: Hashing-based image retrieval. IEEE Transactions on Medical Imaging, 34(2), 496–506.

  • Zhang, X., Xing, F., Su, H., Yang, L., & Zhang, S. (2015c). High-throughput histopathological image analysis via robust cell segmentation and hashing. Medical image analysis, 26(1), 306–315.

  • Zhou, Z., Liu, X., Long, B., & Peng, H. (2016). TReMAP: automatic 3D neuron reconstruction based on tracing, reverse mapping and assembling of 2D projections. Neuroinformatics, 14(1), 41–50.

    Article  PubMed  Google Scholar 

  • Zhou, Z., Sorensen, S., Zeng, H., Hawrylycz, M., & Peng, H. (2015). Adaptive image enhancement for tracing 3D morphologies of neurons and brain vasculatures. Neuroinformatics, 13(2), 153–166.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This work is partially supported by the National Science Foundation under grant ABI-1661280, ABI-1661289, and CNS-1629913.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of North Carolina at Charlotte, Charlotte, NC, 28223, USA

    Zhongyu Li, Erik Butler, Aidong Lu & Shaoting Zhang

  2. Department of Industrial and Systems Engineering, The State University of New Jersey, Piscataway, NJ, 08854, USA

    Kang Li

  3. School of Electrical Engineering and Computer Science, Washington State University, Pullman, WA, 99164, USA

    Shuiwang Ji

Authors
  1. Zhongyu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Erik Butler
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aidong Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shuiwang Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shaoting Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shaoting Zhang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Z., Butler, E., Li, K. et al. Large-scale Exploration of Neuronal Morphologies Using Deep Learning and Augmented Reality. Neuroinform 16, 339–349 (2018). https://doi.org/10.1007/s12021-018-9361-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12021-018-9361-5

Keywords