Skip to main content

Advertisement

SpringerLink
Log in

A mixed reality framework for microsurgery simulation with visual-tactile perception

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Microsurgery is a general term for surgery combining surgical microscope and specialized precision instruments during operation. Training in microsurgery requires considerable time and training resources. With the rapid development of computer technologies, virtual surgery simulation has gained extensive attention over the past decades. In this work, we take advantage of mixed reality (MR) that creates an interactive environment where physical and digital objects coexist, and present an MR framework for the microsurgery simulation. It enables users to practice anastomosis skills with real microsurgical instruments rather than additional haptic feedback devices that are typically used in virtual reality-based systems, and to view a realistic rendering intra-operative scene at the same time, thus creating an immersive training experience with such a visual-tactile interactive environment. A vision-based tracking system is proposed to simultaneously track microsurgical instruments and artificial blood vessels, and a learning-based anatomical modeling approach is introduced to facilitate the development of simulations in different microsurgical specialities by rapidly creating virtual assets. Moreover, we build a prototype system for the simulation specializing in microvascular hepatic artery reconstruction to demonstrate the feasibility and applicability of our framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Tamai, S.: History of microsurgery. Plast. Reconstr. Surg. 124(6S), e282–e294 (2009)

    Google Scholar 

  2. Microsurgeon.org: Microsurgery definition. https://www.microsurgeon.org/microsurgerydef, (2021) Accessed on 08 Apr 2021

  3. Roohi, S.A.: Tips and tricks in microvascular anastomoses. In: The Current Perspectives on Coronary Artery Bypass Grafting. IntechOpen (2020)

  4. Chan, W.Y., Matteucci, P., Southern, S.J.: Validation of microsurgical models in microsurgery training and competence: a review. Microsurg. Off. J. Int. Microsurg. Soc. Eur. Fed. Soc. Microsurg. 27(5), 494–499 (2007)

    Google Scholar 

  5. Byvaltsev, V.A., Akshulakov, S.K., Polkin, R.A., Ochkal, S.V., Stepanov, I.A., Makhambetov, Y.T., Kerimbayev, T.T., Staren, M., Belykh, E., Preul, M.C.: Microvascular anastomosis training in neurosurgery: A review. Minim. Invasive Surg. (2018). https://doi.org/10.1155/2018/6130286

    Article  Google Scholar 

  6. Javid, P., Aydın, A., Mohanna, P.N., Dasgupta, P., Ahmed, K.: Current status of simulation and training models in microsurgery: a systematic review. Microsurgery 39(7), 655–668 (2019)

    Google Scholar 

  7. Crouch, G., Wong, G., Hong, J., Varey, A., Haddad, R., Wang, Z.Z., Wykes, J., Koutalistras, N., Clark, J.R., Solomon, M., et al.: Validated specialty-specific models for multi-disciplinary microsurgery training laboratories: a systematic review. ANZ J. Surg. 91(6), 1110–1116 (2021)

    Google Scholar 

  8. Hosain, M.: Microsurgery training: are the live models era coming in to the tail?–a literature review. Journal of Advances in Medicine and Medical Research pp. 97–109 (2021)

  9. StandfordMedicine: Microsurgery essentials. https://plasticsurgery.stanford.edu/education/microsurgery.html, (2021) Accessed 20 Aug 2021

  10. Wanderer, S., Waltenspuel, C., Grüter, B.E., Strange, F., Sivanrupan, S., Remonda, L., Widmer, H.R., Casoni, D., Andereggen, L., Fandino, J., et al.: Arterial pouch microsurgical bifurcation aneurysm model in the rabbit. JoVE J. Vis. Exp. 159, e61157 (2020)

    Google Scholar 

  11. Erel, E., Aiyenibe, B., Butler, P.E.: Microsurgery simulators in virtual reality. Microsurg. Offi. J. Int. Microsurg. Soc. Eur. Fed. Soc. Microsurg. 23(2), 147–152 (2003)

    Google Scholar 

  12. Hoshyarmanesh, H., Zareinia, K., Lama, S., Sutherland, G.R.: Structural design of a microsurgery-specific haptic device: neuroarmplushd prototype. Mechatronics 73, 102481 (2021)

    Google Scholar 

  13. Rangarajan, K., Davis, H., Pucher, P.H.: Systematic review of virtual haptics in surgical simulation: A valid educational tool? J. Surg. Educ. 77(2), 337–347 (2020)

    Google Scholar 

  14. Yang, L., Etsuko, K.: Review on vision-based tracking in surgical navigation. IET Cyber-Syst. Robot. 2(3), 107–121 (2020)

    Google Scholar 

  15. Xiang, N., Yang, X., Zhang, J.J.: Tsfps: An accurate and flexible 6dof tracking system with fiducial platonic solids. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 4454–4462 (2021)

  16. Neal, M.L., Kerckhoffs, R.: Current progress in patient-specific modeling. Brief. Bioinform. 11(1), 111–126 (2010)

    Google Scholar 

  17. Bücking, T.M., Hill, E.R., Robertson, J.L., Maneas, E., Plumb, A.A., Nikitichev, D.I.: From medical imaging data to 3d printed anatomical models. PloS one 12(5), e0178540 (2017)

    Google Scholar 

  18. Birbara, N.S., Otton, J.M., Pather, N.: 3d modelling and printing technology to produce patient-specific 3d models. Heart Lung Circ. 28(2), 302–313 (2019)

    Google Scholar 

  19. Fanua, S.P., Kim, J., Shaw Wilgis, E.: Alternative model for teaching microsurgery. Microsurgery 21(8), 379–382 (2001)

    Google Scholar 

  20. Atlan, M., Lellouch, A.G., Legagneux, J., Chaouat, M., Masquelet, A.C., Letourneur, D.: A new synthetic model for microvascular anastomosis training? A randomized comparative study between silicone and polyvinyl alcohol gelatin tubes. J. Surg. Educ. 75(1), 182–187 (2018)

    Google Scholar 

  21. Xiao, Z., Samii, M., Wang, J., Pan, Q., Xu, Z., Ju, H.: Training model for the intraluminal continuous suturing technique for microvascular anastomosis. Sci. Rep. 11(1), 1–9 (2021)

    Google Scholar 

  22. Remie, R.: The pvc-rat and other alternatives in microsurgical training. Lab. Anim. 30(9), 48–52 (2001)

    Google Scholar 

  23. De Virgilio, A., Costantino, A., Ebm, C., Conti, V., Mondello, T., Di Bari, M., Cugini, G., Mercante, G., Spriano, G.: High definition three-dimensional exoscope (vitom 3d) for microsurgery training: a preliminary experience. Eur. Arch. Otorhinolaryngol. 277(9), 2589–2595 (2020)

    Google Scholar 

  24. Chen, W.F., Eid, A., Yamamoto, T., Keith, J., Nimmons, G.L., Lawrence, W.T.: A novel supermicrosurgery training model: the chicken thigh. J. Plast. Reconst. Aesthet. Surg. 67(7), 973–978 (2014)

    Google Scholar 

  25. Creighton, F.X., Feng, A.L., Goyal, N., Emerick, K., Deschler, D.: Chicken thigh microvascular training model improves resident surgical skills. Laryngoscope Investig. Otolaryngol. 2(6), 471–474 (2017)

    Google Scholar 

  26. de Giacomo Carneiro, C., Scapini, F.: The rabbit as an experimental model in laryngology (2009)

  27. Yamamoto, T., Yamamoto, N., Yamashita, M., Furuya, M., Hayashi, A., Koshima, I.: Establishment of supermicrosurgical lymphaticovenular anastomosis model in rat. Microsurgery 37(1), 57–60 (2017)

    Google Scholar 

  28. Rodríguez, A., Álvarez, Á., Aguirrezabalaga, J., Martelo, F.: The anteromedial thigh flap as a training model of a perforator flap in rat. J. Reconstr. Microsurg. 23(05), 251–255 (2007)

    Google Scholar 

  29. Otoole, R.V., Playter, R.R., Krummel, T.M., Blank, W.C., Cornelius, N.H., Roberts, W.R., Bell, W.J., Raibert, M.: Measuring and developing suturing technique with a virtual reality surgical simulator. J. Am. Coll. Surg. 189(1), 114–127 (1999)

    Google Scholar 

  30. Kazemi, H., Rappel, J.K., Poston, T., Hai Lim, B., Burdet, E., Leong Teo, C.: Assessing suturing techniques using a virtual reality surgical simulator. Microsurgery 30(6), 479–486 (2010)

    Google Scholar 

  31. Alaraj, A., Luciano, C.J., Bailey, D.P., Elsenousi, A., Roitberg, B.Z., Bernardo, A., Banerjee, P.P., Charbel, F.T.: Virtual reality cerebral aneurysm clipping simulation with real-time haptic feedback. Oper. Neurosurg. 11(1), 52–58 (2015)

    Google Scholar 

  32. Ryu, J., Choi, J., Kim, H.C.: Endoscopic vision-based tracking of multiple surgical instruments during robot-assisted surgery. Artif. Organs 37(1), 107–112 (2013)

    Google Scholar 

  33. Yang, L., Wang, J., Ando, T., Kubota, A., Yamashita, H., Sakuma, I., Chiba, T., Kobayashi, E.: Vision-based endoscope tracking for 3d ultrasound image-guided surgical navigation. Comput. Med. Imaging Graph. 40, 205–216 (2015)

    Google Scholar 

  34. Sánchez-González, P., Cano, A.M., Oropesa, I., Sánchez-Margallo, F.M., Pozo, F.D., Lamata, P., Gómez, E.J.: Laparoscopic video analysis for training and image-guided surgery. Minim. Invasive Ther. Allied Technol. 20(6), 311–320 (2011)

    Google Scholar 

  35. Robu, M., Kadkhodamohammadi, A., Luengo, I., Stoyanov, D.: Towards real-time multiple surgical tool tracking. Comput. Methods Biomech. Biomed. Eng. Imaging Vis. pp. 1–7 (2020)

  36. Liu, X., Plishker, W., Shekhar, R.: Hybrid electromagnetic-aruco tracking of laparoscopic ultrasound transducer in laparoscopic video. J. Med. Imaging 8(1), 015001 (2021)

    Google Scholar 

  37. Gadwe, A., Ren, H.: Real-time 6dof pose estimation of endoscopic instruments using printable markers. IEEE Sens. J. 19(6), 2338–2346 (2018)

    Google Scholar 

  38. Richa, R., Poignet, P., Liu, C.: Three-dimensional motion tracking for beating heart surgery using a thin-plate spline deformable model. Int. J. Robot. Res. 29(2–3), 218–230 (2010)

    Google Scholar 

  39. Wong, W.K., Yang, B., Liu, C., Poignet, P.: A quasi-spherical triangle-based approach for efficient 3-d soft-tissue motion tracking. IEEE/ASME Trans. Mechatron. 18(5), 1472–1484 (2012)

    Google Scholar 

  40. Yang, B., Wong, W.K., Liu, C., Poignet, P.: 3d soft-tissue tracking using spatial-color joint probability distribution and thin-plate spline model. Pattern Recogn. 47(9), 2962–2973 (2014)

    MATH  Google Scholar 

  41. Schoob, A., Kundrat, D., Kahrs, L.A., Ortmaier, T.: Stereo vision-based tracking of soft tissue motion with application to online ablation control in laser microsurgery. Med. Image Anal. 40, 80–95 (2017)

    Google Scholar 

  42. Song, J., Wang, J., Zhao, L., Huang, S., Dissanayake, G.: Dynamic reconstruction of deformable soft-tissue with stereo scope in minimal invasive surgery. IEEE Robot. Autom. Lett. 3(1), 155–162 (2017)

    Google Scholar 

  43. Antiga, L., Piccinelli, M., Botti, L., Ene-Iordache, B., Remuzzi, A., Steinman, D.A.: An image-based modeling framework for patient-specific computational hemodynamics. Med. Biol. Eng. Comput. 46(11), 1097–1112 (2008)

    Google Scholar 

  44. Taylor, C.A., Figueroa, C.: Patient-specific modeling of cardiovascular mechanics. Annu. Rev. Biomed. Eng. 11, 109–134 (2009)

    Google Scholar 

  45. Tahir, W., Kura, S., Zhu, J., Cheng, X., Damseh, R., Tadesse, F., Seibel, A., Lee, B.S., Lesage, F., Sakadžic, S., et al.: Anatomical modeling of brain vasculature in two-photon microscopy by generalizable deep learning. BME Frontiers 2021 (2021)

  46. Bhalodia, R., Elhabian, S.Y., Kavan, L., Whitaker, R.T.: Deepssm: A deep learning framework for statistical shape modeling from raw images. In: International Workshop on Shape in Medical Imaging, pp. 244–257. Springer (2018)

  47. Kong, F., Wilson, N., Shadden, S.C.: A deep-learning approach for direct whole-heart mesh reconstruction. arXiv preprint arXiv:2102.07899 (2021)

  48. Müller, M., Heidelberger, B., Hennix, M., Ratcliff, J.: Position based dynamics. J. Vis. Commun. Image Represent. 18(2), 109–118 (2007)

    Google Scholar 

  49. Liu, M.Y., Tuzel, O., Veeraraghavan, A., Chellappa, R.: Fast directional chamfer matching. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1696–1703. IEEE (2010)

  50. Yu, Q., Wei, H., Yang, C.: Local part chamfer matching for shape-based object detection. Pattern Recogn. 65, 82–96 (2017)

    Google Scholar 

  51. Hartley, R.I., Sturm, P.: Triangulation. Computer vision and image understanding 68(2), 146–157 (1997)

  52. Liu, S., Li, T., Chen, W., Li, H.: Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7708–7717 (2019)

  53. Xiang, N., Wang, R., Jiang, T., Wang, L., Li, Y., Yang, X., Zhang, J.: Sketch-based modeling with a differentiable renderer. Comput. Anim. Virtual Worlds 31(4–5), e1939 (2020)

    Google Scholar 

  54. Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)

  55. Kato, H., Ushiku, Y., Harada, T.: Neural 3d mesh renderer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3907–3916 (2018)

  56. WirelessWorld: Advantages of x-ray | disadvantages of x-ray. https://www.rfwireless-world.com/Terminology/Advantages-and-Disadvantages-of-X-Ray.html (2021), Accessed on 15 Aug 2021

  57. Pharr, M., Jakob, W., Humphreys, G.: Physically based rendering: from theory to implementation. Morgan Kaufmann (2016)

  58. Xu, L., Liu, Q.: Real-time inextensible surgical thread simulation. Int. J. Comput. Assist. Radiol. Surg. 13(7), 1019–1035 (2018)

    Google Scholar 

  59. Yu, P., Pan, J., Qin, H., Hao, A., Wang, H.: Real-time suturing simulation for virtual reality medical training. Comput. Anim. Virtual Worlds 31(4–5), e1940 (2020)

    Google Scholar 

  60. Wu, P.C., Wang, R., Kin, K., Twigg, C., Han, S., Yang, M.H., Chien, S.Y.: Dodecapen: Accurate 6dof tracking of a passive stylus. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 365–374 (2017)

  61. Senior, M.A., Southern, S.J., Majumder, S.: Microvascular simulator-a device for micro-anastomosis training. Ann. R. Coll. Surg. Engl. 83(5), 358 (2001)

    Google Scholar 

  62. Brown, J., Sorkin, S., Latombe, J.C., Montgomery, K., Stephanides, M.: Algorithmic tools for real-time microsurgery simulation. Med. Image Anal. 6(3), 289–300 (2002)

    Google Scholar 

  63. Grober, E.D., Hamstra, S.J., Wanzel, K.R., Reznick, R.K., Matsumoto, E.D., Sidhu, R.S., Jarvi, K.A.: Validation of novel and objective measures of microsurgical skill: hand-motion analysis and stereoscopic visual acuity. Microsurgery 23(4), 317–322 (2003)

    Google Scholar 

Download references

Acknowledgements

This work was supported by XJTLU Research Development Fund No. RDF-21-02-065 and Neuravatar Project funded by High Education Innovation Fund (UK-HEIF).

Author information

Authors and Affiliations

  1. Department of Computing, School of Advanced Technology, Xi’an Jiaotong-Liverpool University, Suzhou, China

    Nan Xiang, Hai-Ning Liang & Lingyun Yu

  2. National Centre for Computer Animation, Bournemouth University, Poole, UK

    Xiaosong Yang & Jian J. Zhang

Authors
  1. Nan Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hai-Ning Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lingyun Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaosong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jian J. Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Nan Xiang or Xiaosong Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiang, N., Liang, HN., Yu, L. et al. A mixed reality framework for microsurgery simulation with visual-tactile perception. Vis Comput 39, 3661–3673 (2023). https://doi.org/10.1007/s00371-023-02964-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02964-1

Keywords

Search

Navigation