Skip to main content
Springer Nature Link
Log in

Fireflies: Photorealistic Simulation and Optimization of Structured Light Endoscopy

  • Conference paper
  • First Online:
Simulation and Synthesis in Medical Imaging (SASHIMI 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15187))

Included in the following conference series:

  • 237 Accesses

Abstract

There exist various approaches for the 3D reconstruction of dynamic scenes. In medicine, particularly in endoscopy, single-shot structured light systems are frequently explored, as they allow for the reconstruction of dynamic, feature-less surfaces. Design and manufacturing of structured light endoscopes, however, implies high initial costs that significantly hinder the availability and development of these systems. To streamline this process, simulation systems are necessary that allow researchers to not only model the intricacies of medical domains, but also of structured light systems themselves. To address this, we propose Fireflies, a differentiable framework for the physically-based simulation and domain randomization of structured light endoscopy. Based on the differentiable Mitsuba renderer, Fireflies facilitates and simplifies the development of domain-specific algorithms for endoscopic procedures. In this paper, we demonstrate the effectiveness of our framework by jointly optimizing domain-specific laser-based projection pattern for Structured Light Endoscopy, and generating large-scale synthetic training data for efficient supervised learning without manual labeling. We show that a) an optimized projection pattern can increase the reconstructability of a target domain and b) the synthetic data generated by Fireflies lowers the labeling effort required for endoscopic machine learning tasks. The source code is available at: https://github.com/Henningson/Fireflies

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M., Kalinin, A.A.: Albumentations: fast and flexible image augmentations. Information 11(2),  125 (2020). https://doi.org/10.3390/info11020125

    Article  Google Scholar 

  2. Christoph Heindl, Lukas Brunner, S.Z., Scharinger, J.: Blendtorch: A real-time, adaptive domain randomization library. In: ICPR (2020)

    Google Scholar 

  3. Denninger, M., Winkelbauer, D., Sundermeyer, M., Boerdijk, W., Knauer, M., Strobl, K.H., Humt, M., Triebel, R.: Blenderproc2: A procedural pipeline for photorealistic rendering. JOSS 8(82),  4901 (2023). https://doi.org/10.21105/joss.04901

  4. Furukawa, R., Oka, S., Kotachi, T., Okamoto, Y., Tanaka, S., Sagawa, R., Kawasaki, H.: Fully auto-calibrated active-stereo-based 3d endoscopic system using correspondence estimation with graph convolutional network. In: IEEE EMBC. pp. 4357–4360. IEEE (2020). https://doi.org/10.1109/EMBC44109.2020.9176417

  5. Furukawa, R., Sagawa, R., Oka, S., Tanaka, S., Kawasaki, H.: Single and multi-frame auto-calibration for 3d endoscopy with differential rendering. In: IEEE EMBC. pp. 1–5. IEEE (2023). https://doi.org/10.1109/EMBC40787.2023.10340381

  6. Goceri, E.: Medical image data augmentation: techniques, comparisons and interpretations. Artif. Intell. Rev. 56(11), 12561–12605 (2023). https://doi.org/10.1007/s10462-023-10453-z

    Article  Google Scholar 

  7. Henningson, J.O., Semmler, M., Döllinger, M., Stamminger, M.: Joint segmentation and sub-pixel localization in structured light laryngoscopy. In: MICCAI. pp. 34–43. Springer (2023). https://doi.org/10.1007/978-3-031-43987-2_4

  8. Henningson, J.O., Stamminger, M., Döllinger, M., Semmler, M.: Real-time 3d reconstruction of human vocal folds via high-speed laser-endoscopy. In: MICCAI. pp. 3–12. Springer (2022). https://doi.org/10.1007/978-3-031-16449-1_1

  9. İncetan, K., Celik, I.O., Obeid, A., Gokceler, G.I., Ozyoruk, K.B., Almalioglu, Y., Chen, R.J., Mahmood, F., Gilbert, H., Durr, N.J., et al.: Vr-caps: a virtual environment for capsule endoscopy. MIA 70, 101990 (2021). https://doi.org/10.1016/j.media.2021.101990

    Article  Google Scholar 

  10. Kist, A.M., Zilker, J., Döllinger, M., Semmler, M.: Feature-based image registration in structured light endoscopy. In: MIDL. pp. 369–383. PMLR (2021)

    Google Scholar 

  11. Li, T., Bolkart, T., Black, M.J., Li, H., Romero, J.: Learning a model of facial shape and expression from 4d scans. ACM ToG 36(6), 194–1 (2017). https://doi.org/10.1145/3130800.3130813

    Article  Google Scholar 

  12. Luegmair, G., Mehta, D.D., Kobler, J.B., Döllinger, M.: Three-dimensional optical reconstruction of vocal fold kinematics using high-speed video with a laser projection system. IEEE TMI 34(12), 2572–2582 (2015). https://doi.org/10.1109/TMI.2015.2445921

    Article  Google Scholar 

  13. Maier-Hein, L., Groch, A., Bartoli, A., Bodenstedt, S., Boissonnat, G., Chang, P.L., Clancy, N.T., Elson, D.S., Haase, S., Heim, E., et al.: Comparative validation of single-shot optical techniques for laparoscopic 3-d surface reconstruction. IEEE TMI 33(10), 1913–1930 (2014). https://doi.org/10.1109/TMI.2014.2325607

    Article  Google Scholar 

  14. Maier-Hein, L., Mountney, P., Bartoli, A., Elhawary, H., Elson, D., Groch, A., Kolb, A., Rodrigues, M., Sorger, J., Speidel, S., et al.: Optical techniques for 3d surface reconstruction in computer-assisted laparoscopic surgery. MIA 17(8), 974–996 (2013). https://doi.org/10.1016/j.media.2013.04.003

    Article  Google Scholar 

  15. Maurice, X., Albitar, C., Doignon, C., de Mathelin, M.: A structured light-based laparoscope with real-time organs’ surface reconstruction for minimally invasive surgery. In: IEEE EMBC. pp. 5769–5772. IEEE (2012). https://doi.org/10.1109/EMBC.2012.6347305

  16. Mihai, D., Hare, J.: Differentiable drawing and sketching. arXiv preprint arXiv:2103.16194 (2021). https://doi.org/10.48550/arXiv.2103.16194

  17. Mikamo, M., Furukawa, R., Oka, S., Kotachi, T., Okamoto, Y., Tanaka, S., Sagawa, R., Kawasaki, H.: Active stereo method for 3d endoscopes using deep-layer gcn and graph representation with proximity information. In: IEEE EMBC. pp. 7551–7555. IEEE (2021). https://doi.org/10.1109/EMBC46164.2021.9629696

  18. Motamed, S., Rogalla, P., Khalvati, F.: Data augmentation using generative adversarial networks (gans) for gan-based detection of pneumonia and covid-19 in chest x-ray images. IMU 27, 100779 (2021). https://doi.org/10.1016/j.imu.2021.100779

    Article  Google Scholar 

  19. Nimier-David, M., Vicini, D., Zeltner, T., Jakob, W.: Mitsuba 2: A retargetable forward and inverse renderer. ACM ToG 38(6), 1–17 (2019). https://doi.org/10.1145/3355089.3356498

    Article  Google Scholar 

  20. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. NeurIPS 32 (2019)

    Google Scholar 

  21. Patel, R.R., Donohue, K.D., Lau, D., Unnikrishnan, H.: In vivo measurement of pediatric vocal fold motion using structured light laser projection. JOV 27(4), 463–472 (2013). https://doi.org/10.1016/j.jvoice.2013.03.004

    Article  Google Scholar 

  22. Pérez-García, F., Sparks, R., Ourselin, S.: Torchio: a python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning. CMPB 208, 106236 (2021). https://doi.org/10.1016/j.cmpb.2021.106236

    Article  Google Scholar 

  23. Rakhimov, R., Ardelean, A.T., Lempitsky, V., Burnaev, E.: Npbg++: Accelerating neural point-based graphics. In: IEEE CVPR. pp. 15969–15979 (2022). https://doi.org/10.1109/CVPR52688.2022.01550

  24. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI. pp. 234–241. Springer (2015). https://doi.org/10.1007/978-3-319-24574-4_28

  25. Rückert, D., Franke, L., Stamminger, M.: Adop: Approximate differentiable one-pixel point rendering. ACM ToG 41(4), 1–14 (2022). https://doi.org/10.1145/3528223.3530122

    Article  Google Scholar 

  26. Semmler, M., Kniesburges, S., Birk, V., Ziethe, A., Patel, R., Döllinger, M.: 3d reconstruction of human laryngeal dynamics based on endoscopic high-speed recordings. IEEE TMI 35(7), 1615–1624 (2016). https://doi.org/10.1109/TMI.2016.2521419

    Article  Google Scholar 

  27. Semmler, M., Kniesburges, S., Parchent, J., Jakubaß, B., Zimmermann, M., Bohr, C., Schützenberger, A., Döllinger, M.: Endoscopic laser-based 3d imaging for functional voice diagnostics. Appl. Sci. 7(6),  600 (2017). https://doi.org/10.3390/app7060600

    Article  Google Scholar 

  28. Visentini-Scarzanella, M., Kawasaki, H., Furukawa, R., Bonino, M.A., Arolfo, S., Secco, G.L., Arezzo, A., Menciassi, A., Dario, P., Ciuti, G.: A structured light laser probe for gastrointestinal polyp size measurement: a preliminary comparative study. EIO 6(05), E602–E609 (2018). https://doi.org/10.1055/a-0577-2798

    Article  Google Scholar 

  29. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: ICCV. pp. 4471–4480 (2019). https://doi.org/10.1109/iccv.2019.00457

Download references

Acknowledgements

We thank Moritz Kappel for valuable feedback. This work was supported by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant STA662/6-1, Project-ID 448240908 and (partly) funded by the DFG - SFB 1483 - Project-ID 442419336, EmpkinS. The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center of the Friedrich-Alexander-Universität Erlangen-Nürnberg.

Author information

Authors and Affiliations

  1. Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

    Jann-Ole Henningson & Marc Stamminger

  2. Division of Phoniatrics and Pediatric Audiology at the Department of Otorhinolaryngology, Head and Neck Surgery, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, 91054, Erlangen, Germany

    Reinhard Veltrup, Marion Semmler & Michael Döllinger

Authors
  1. Jann-Ole Henningson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Reinhard Veltrup
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marion Semmler
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Döllinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marc Stamminger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jann-Ole Henningson .

Editor information

Editors and Affiliations

  1. King's College London, London, UK

    Virginia Fernandez

  2. University of Twente, Enschede, Gelderland, The Netherlands

    Jelmer M. Wolterink

  3. Masaryk University, Brno, Czech Republic

    David Wiesner

  4. Johns Hopkins University, Baltimore, MD, USA

    Samuel Remedios

  5. Vanderbilt University, Nashville, TN, USA

    Lianrui Zuo

  6. University of Girona, Girona, Spain

    Adrià Casamitjana

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Henningson, JO., Veltrup, R., Semmler, M., Döllinger, M., Stamminger, M. (2025). Fireflies: Photorealistic Simulation and Optimization of Structured Light Endoscopy. In: Fernandez, V., Wolterink, J.M., Wiesner, D., Remedios, S., Zuo, L., Casamitjana, A. (eds) Simulation and Synthesis in Medical Imaging. SASHIMI 2024. Lecture Notes in Computer Science, vol 15187. Springer, Cham. https://doi.org/10.1007/978-3-031-73281-2_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-73281-2_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-73280-5

  • Online ISBN: 978-3-031-73281-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships