Abstract
High-speed optical 3-D fluorescence microscopy is an essential tool for capturing the rapid dynamics of biological systems such as cellular signaling and complex movements. Designing such an optical system is constrained by the inherent trade-off among resolution, speed, and noise which comes from the limited number of photons that can be collected. In this paper, we propose a recursive light propagation network (RLP-Net) that infers the 3-D volume from two adjacent 2-D wide-field fluorescence images via virtual refocusing. Specifically, we propose a recursive inference scheme in which the network progressively predicts the subsequent planes along the axial direction. This recursive inference scheme reflects that the law of physics for the light propagation remains spatially invariant and therefore a fixed function (i.e., a neural network) for a short distance light propagation can be recursively applied for a longer distance light propagation. Experimental results show that the proposed method can faithfully reconstruct the 3-D volume from two planes in terms of both quantitative measures and visual quality. The source code used in the paper is available at https://github.com/NICALab/rlpnet.
C. Shin and H. Ryu—Equal contributions.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Abrahamsson, S., et al.: Fast multicolor 3D imaging using aberration-corrected multifocus microscopy. Nat. Methods 10, 60–63 (2013)
Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41–48 (2009)
Bouchard, M.B., et al.: Swept confocally-aligned planar excitation (SCAPE) microscopy for high-speed volumetric imaging of behaving organisms. Nat. Photonics 9(2), 113–119 (2015)
Chen, Q., et al.: Imaging neural activity using Thy1-GCaMP transgenic mice. Neuron 76(2), 297–308 (2012)
Chen, T.-W., et al.: Ultrasensitive fluorescent proteins for imaging neuronal activity. Nature 499(7458), 295–300 (2013)
Cong, L., et al.: Rapid whole brain imaging of neural activity in freely behaving larval zebrafish (Danio rerio). eLife 6, e28158 (2017)
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134. IEEE (2017)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations, pp. 1–15 (2015)
Mao, X., Shen, C., Yang, Y.-B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections, pp. 2810–2818 (2016)
Mechrez, R., Talmi, I., Shama, F., Zelnik-Manor, L.: Maintaining natural image statistics with the contextual loss. arXiv:1803.04626 (2018)
Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv:1411.1784 (2014)
Ounkomol, C., Seshamani, S., Maleckar, M.M., Collman, F., Johnson, G.R.: Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nat. Methods \(\varvec {15}\), 917–920 (2018). https://doi.org/10.1038/s41592-018-0111-2
Pawley, J.: Handbook of Biological Confocal Microscopy, 3rd edn. Springer, Boston (2006). https://doi.org/10.1007/978-0-387-45524-2
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8026–8037 (2019)
Piatkevich, K.D., et al.: A robotic multidimensional directed evolution of proteins: development and application to fluorescent voltage reporters. Nat. Chem. Biol. \(\varvec {14}\), 352–360 (2017)
Prevedel, R., et al.: Simultaneous whole-animal 3D imaging of neuronal activity using light-field microscopy. Nat. Methods \(\varvec {11}\), 727–730 (2014)
Schrödel, T., Prevedel, R., Aumayr, K., Zimmer, M., Vaziri, A.: Brain-wide 3D imaging of neuronal activity in Caenorhabditis elegans with sculpted light. Nat. Methods 10, 1013–1020 (2013). https://doi.org/10.1038/nmeth.2637
Tian, L., et al.: Imaging neural activity in worms, flies and mice with improved GCaMP calcium indicators. Nat. Methods \(\varvec {6}\)(12), 875–881 (2009)
Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 63–79. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-11021-5_5
Wang, Z., et al.: Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning. Nat. Methods (2021). https://doi.org/10.1038/s41592-021-01058-x
Weigert, M., Royer, L., Jug, F., Myers, G.: Isotropic reconstruction of 3D fluorescence microscopy images using convolutional neural networks. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 126–134. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66185-8_15
Wu, Y., et al.: Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning. Nat. Methods \(\varvec {16}\), 1323–1331 (2019). https://doi.org/10.1038/s41592-019-0622-5
Xingjian, S., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.C.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Advances in Neural Information Processing Systems, pp. 802–810 (2015)
Yoon, Y.-G., et al.: Sparse decomposition light-field microscopy for high speed imaging of neuronal activity. Optica \(\varvec {7}\), 1457–1468 (2020)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232. IEEE (2017)
Acknowledgements
This research was supported by National Research Foundation of Korea (2020R1C1C1009869) and the Korea Medical Device Development Fund grant funded by the Korea government (202011B21-05).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 9724 KB)
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Shin, C., Ryu, H., Cho, ES., Yoon, YG. (2021). RLP-Net: A Recursive Light Propagation Network for 3-D Virtual Refocusing. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12906. Springer, Cham. https://doi.org/10.1007/978-3-030-87231-1_18
Download citation
DOI: https://doi.org/10.1007/978-3-030-87231-1_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87230-4
Online ISBN: 978-3-030-87231-1
eBook Packages: Computer ScienceComputer Science (R0)
