Abstract
Effective approaches for accurately predicting the developmental potential of embryos and selecting suitable embryos for blastocyst culture are critically needed. Many deep learning (DL) based methods for time-lapse monitoring (TLM) videos have been proposed to tackle this problem. Although fruitful, these methods are either ineffective when processing long TLM videos, or need extra annotations to determine the morphokinetics parameters of embryos. In this paper, we propose Adaptive Key Frame Selection (AdaKFS), a new framework that adaptively selects informative frames on per-input basis to predict blastocyst formation using TLM videos at the cleavage stage on day 3. For each time step, a policy network decides whether to use or skip the current frame. Further, a prediction network generates prediction using the morphokinetics features of the selected frames. We efficiently train and enhance the frame selection process by using a Gumbel-Softmax sampling approach and a reward function, respectively. Comprehensive experiments on a large TLM video dataset verify the performance superiority of our new method over state-of-the-art methods.
T. Chen and Y. Cheng–Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abbasi, M., Saeedi, P., Au, J., Havelock, J.: A deep learning approach for prediction of IVF implantation outcome from day 3 and day 5 time-lapse human embryo image sequences. In: 2021 IEEE International Conference on Image Processing (ICIP), pp. 289–293. IEEE (2021)
Gardner, D.K., Lane, M., Stevens, J., Schlenker, T., Schoolcraft, W.B.: Blastocyst score affects implantation and pregnancy outcome: towards a single blastocyst transfer. Fertil. Steril. 73(6), 1155–1158 (2000)
Glujovsky, D., Farquhar, C., Retamar, A.M.Q., Sedo, C.R.A., Blake, D.: Cleavage stage versus blastocyst stage embryo transfer in assisted reproductive technology. Cochrane Database Syst. Rev. (6) (2016)
Herrero, J., Tejera, A., Albert, C., Vidal, C., de los Santos, M.J., Meseguer, M.: A time to look back: analysis of morphokinetic characteristics of human embryo development. Fertil. Steril. 100(6), 1602–1609 (2013)
Holte, J., et al.: Construction of an evidence-based integrated morphology cleavage embryo score for implantation potential of embryos scored and transferred on day 2 after oocyte retrieval. Hum. Reprod. 22(2), 548–557 (2007)
Huang, B., et al.: Elevated progesterone levels on the day of oocyte maturation may affect top quality embryo IVF cycles. PLOS ONE 11(1), e0145895 (2016)
Jang, E., Gu, S., Poole, B.: Categorical reparametrization with Gumbel-Softmax. In: Proceedings International Conference on Learning Representations (ICLR), April 2017. https://openreview.net/pdf?id=rkE3y85ee
Khosravi, P., et al.: Deep learning enables robust assessment and selection of human blastocysts after in vitro fertilization. NPJ Digital Med. 2(1), 1–9 (2019)
Kragh, M.F., Rimestad, J., Berntsen, J., Karstoft, H.: Automatic grading of human blastocysts from time-lapse imaging. Comput. Biol. Med. 115, 103494 (2019)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25 (2012)
Leahy, B.D., et al.: Automated measurements of key morphological features of human embryos for IVF. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12265, pp. 25–35. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_3
Lemmen, J., Agerholm, I., Ziebe, S.: Kinetic markers of human embryo quality using time-lapse recordings of IVF/ICSI-fertilized oocytes. Reprod. Biomed. Online 17(3), 385–391 (2008)
Liao, Q., et al.: Development of deep learning algorithms for predicting blastocyst formation and quality by time-lapse monitoring. Commun. Biol. 4(1), 1–9 (2021)
Liu, Z., et al.: Multi-task deep learning with dynamic programming for embryo early development stage classification from time-lapse videos. IEEE Access 7, 122153–122163 (2019)
Lockhart, L., Saeedi, P., Au, J., Havelock, J.: Automating embryo development stage detection in time-lapse imaging with synergic loss and temporal learning. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 540–549. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_52
Lukyanenko, S., et al.: Developmental stage classification of embryos using two-stream neural network with linear-chain conditional random field. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 363–372. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_35
Ma, S., Sigal, L., Sclaroff, S.: Learning activity progression in LSTMs for activity detection and early detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1942–1950 (2016)
Meseguer, M., Herrero, J., Tejera, A., Hilligsøe, K.M., Ramsing, N.B., Remohí, J.: The use of morphokinetics as a predictor of embryo implantation\(\dagger \). Hum. Reprod. 26(10), 2658–2671 (2011). https://doi.org/10.1093/humrep/der256
Motato, Y., de los Santos, M.J., Escriba, M.J., Ruiz, B.A., Remohí, J., Meseguer, M.: Morphokinetic analysis and embryonic prediction for blastocyst formation through an integrated time-lapse system. Fertil. Steril. 105(2), 376–384 (2016)
Papanikolaou, E.G., et al.: Live birth rate is significantly higher after blastocyst transfer than after cleavage-stage embryo transfer when at least four embryos are available on day 3 of embryo culture. a randomized prospective study. Hum. Reprod. 20(11), 3198–3203 (2005)
Tran, D., Cooke, S., Illingworth, P.J., Gardner, D.K.: Deep learning as a predictive tool for fetal heart pregnancy following time-lapse incubation and blastocyst transfer. Hum. Reprod. 34(6), 1011–1018 (2019)
Vaswani, A., et al.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017)
Zeman, A., Maerten, A.-S., Mengels, A., Sharon, L.F., Spiessens, C., de Beeck, Hans Op: Deep learning for human embryo classification at the cleavage stage (Day 3). In: Del Bimbo, A., et al. (eds.) ICPR 2021. LNCS, vol. 12661, pp. 278–292. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-68763-2_21
Acknowledgment
This research was partially supported by National Key R &D Program of China under grant No. 2019YFC0118802, National Natural Science Foundation of China under grants No. 62176231 and No. 62106218, Zhejiang Public Welfare Technology Research Project under grant No. LGF20F020013, Wenzhou Bureau of Science and Technology of China (No. Y2020082). D. Z. Chen’s research was supported in part by NSF Grant CCF-1617735.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, T. et al. (2022). Automating Blastocyst Formation and Quality Prediction in Time-Lapse Imaging with Adaptive Key Frame Selection. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13434. Springer, Cham. https://doi.org/10.1007/978-3-031-16440-8_43
Download citation
DOI: https://doi.org/10.1007/978-3-031-16440-8_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16439-2
Online ISBN: 978-3-031-16440-8
eBook Packages: Computer ScienceComputer Science (R0)
