Skip to main content
Springer Nature Link
Log in

Large-Scale Multi-hypotheses Cell Tracking Using Ultrametric Contours Maps

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15137))

Included in the following conference series:

  • 359 Accesses

Abstract

In this work, we describe a method for large-scale 3D cell-tracking through a segmentation selection approach. The proposed method is effective at tracking cells across large microscopy datasets on two fronts: (i) It can solve problems containing millions of segmentation instances in terabyte-scale 3D+t datasets; (ii) It achieves competitive results with or without deep learning, bypassing the requirement of 3D annotated data, that is scarce in the fluorescence microscopy field. The proposed method computes cell tracks and segments using a hierarchy of segmentation hypotheses and selects disjoint segments by maximizing the overlap between adjacent frames. We show that this method is the first to achieve state-of-the-art in both nuclei- and membrane-based cell tracking by evaluating it on the 2D epithelial cell benchmark and 3D images from the cell tracking challenge. Furthermore, it has a faster integer linear programming formulation, and the framework is flexible, supporting segmentations from individual off-the-shelf cell segmentation models or their combination as an ensemble. The code is available as supplementary material.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/funkey/flywing.

References

  1. Aigouy, B., Umetsu, D., Eaton, S.: Segmentation and quantitative analysis of epithelial tissues. In: Drosophila: Methods and Protocols, pp. 227–239 (2016)

    Google Scholar 

  2. Akbaş, C.E., Ulman, V., Maška, M., Jug, F., Kozubek, M.: Automatic fusion of segmentation and tracking labels. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11134, pp. 446–454. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11024-6_34

    Chapter  Google Scholar 

  3. Amat, F., et al.: Fast, accurate reconstruction of cell lineages from large-scale fluorescence microscopy data. Nat. Methods 11(9), 951–958 (2014)

    Article  Google Scholar 

  4. Arbelaez, P.: Boundary extraction in natural images using ultrametric contour maps. In: Conference on Computer Vision and Pattern Recognition Workshop, pp. 182–182. IEEE (2006)

    Google Scholar 

  5. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2010)

    Article  Google Scholar 

  6. Bailoni, A., et al.: GASP, a generalized framework for agglomerative clustering of signed graphs and its application to instance segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 11645–11655 (2022)

    Google Scholar 

  7. Ben-Haim, T., Raviv, T.R.: Graph neural network for cell tracking in microscopy videos. In: European Conference on Computer Vision, pp. 610–626. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-19803-8_36

  8. Berg, S., et al.: Ilastik: interactive machine learning for (bio) image analysis. Nat. Methods 16(12), 1226–1232 (2019)

    Article  Google Scholar 

  9. Bise, R., Maeda, Y., Kim, M.h., Kino-Oka, M.: Cell tracking under high confluency conditions by candidate cell region detection-based-association approach. Biomed. Eng. 1004–1010 (2013)

    Google Scholar 

  10. Briggman, K., Denk, W., Seung, S., Helmstaedter, M., Turaga, S.C.: Maximin affinity learning of image segmentation, vol. 22 (2009)

    Google Scholar 

  11. Couprie, C., Grady, L., Najman, L., Talbot, H.: Power watersheds: a new image segmentation framework extending graph cuts, random walker and optimal spanning forest. In: IEEE International Conference on Computer Vision, pp. 731–738 (2009)

    Google Scholar 

  12. Cousty, J., Bertrand, G., Najman, L., Couprie, M.: Watershed cuts: minimum spanning forests and the drop of water principle. IEEE Trans. Pattern Anal. Mach. Intell. 31(8), 1362–1374 (2009)

    Article  Google Scholar 

  13. Deng, R., Shen, C., Liu, S., Wang, H., Liu, X.: Learning to predict crisp boundaries. In: European Conference on Computer Vision, pp. 562–578 (2018)

    Google Scholar 

  14. Dzyubachyk, O., Van Cappellen, W.A., Essers, J., Niessen, W.J., Meijering, E.: Advanced level-set-based cell tracking in time-lapse fluorescence microscopy. IEEE Trans. Med. Imaging 29(3), 852–867 (2010)

    Article  Google Scholar 

  15. Friedman, J.H.: Stochastic gradient boosting. Comput. Stat. Data Anal. 38(4), 367–378 (2002)

    Article  MathSciNet  Google Scholar 

  16. Funke, J., Mais, L., Champion, A., Dye, N., Kainmueller, D.: A benchmark for epithelial cell tracking. In: Proceedings of The European Conference on Computer Vision (ECCV) Workshops (2018)

    Google Scholar 

  17. Funke, J., et al.: Large scale image segmentation with structured loss based deep learning for connectome reconstruction. IEEE Trans. Pattern Anal. Mach. Intell. 41(7), 1669–1680 (2018)

    Article  Google Scholar 

  18. Hayashida, J., Bise, R.: Cell tracking with deep learning for cell detection and motion estimation in low-frame-rate. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 397–405. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_44

    Chapter  Google Scholar 

  19. Hayashida, J., Nishimura, K., Bise, R.: Mpm: joint representation of motion and position map for cell tracking. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3823–3832 (2020)

    Google Scholar 

  20. Hirsch, P., Malin-Mayor, C., Santella, A., Preibisch, S., Kainmueller, D., Funke, J.: Tracking by weakly-supervised learning and graph optimization for whole-embryo c. elegans lineages. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 25–35. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-16440-8_3

  21. Huisken, J., Swoger, J., Del Bene, F., Wittbrodt, J., Stelzer, E.H.: Optical sectioning deep inside live embryos by selective plane illumination microscopy. Science 305(5686), 1007–1009 (2004)

    Article  Google Scholar 

  22. Januszewski, M., et al.: High-precision automated reconstruction of neurons with flood-filling networks. Nat. Methods 15(8), 605–610 (2018)

    Article  Google Scholar 

  23. Jug, F., Levinkov, E., Blasse, C., Myers, E.W., Andres, B.: Moral lineage tracing. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5926–5935 (2016)

    Google Scholar 

  24. Kappes, J.H., Speth, M., Andres, B., Reinelt, G., Schn, C.: Globally optimal image partitioning by multicuts. In: Boykov, Y., Kahl, F., Lempitsky, V., Schmidt, F.R. (eds.) EMMCVPR 2011. LNCS, vol. 6819, pp. 31–44. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23094-3_3

    Chapter  Google Scholar 

  25. Keller, P.J., Schmidt, A.D., Wittbrodt, J., Stelzer, E.H.: Reconstruction of zebrafish early embryonic development by scanned light sheet microscopy. Science 322(5904), 1065–1069 (2008)

    Article  Google Scholar 

  26. Keuper, M., Levinkov, E., Bonneel, N., Lavoué, G., Brox, T., Andres, B.: Efficient decomposition of image and mesh graphs by lifted multicuts. In: IEEE International Conference on Computer Vision, pp. 1751–1759 (2015)

    Google Scholar 

  27. Kiran, B.R., Serra, J.: Global-local optimizations by hierarchical cuts and climbing energies. Pattern Recogn. 47(1), 12–24 (2014)

    Article  Google Scholar 

  28. Lotufo, R.D.A., Falcão, A.X., Zampirolli, F.A.: IFT-watershed from gray-scale marker. In: Conference on Graphics, Patterns and Images (SIBGRAPI), pp. 146–152. IEEE (2002)

    Google Scholar 

  29. Löffler, K., Mikut, R.: Embedtrack–simultaneous cell segmentation and tracking through learning offsets and clustering bandwidths. IEEE Access 10, 77147–77157 (2022). https://doi.org/10.1109/ACCESS.2022.3192880

    Article  Google Scholar 

  30. Magnusson, K.E.: Segmentation and tracking of cells and particles in time-lapse microscopy. Ph.D. thesis, KTH Royal Institute of Technology (2016)

    Google Scholar 

  31. Malin-Mayor, C., et al.: Automated reconstruction of whole-embryo cell lineages by learning from sparse annotations. Nat. Biotechnol. 1–6 (2022)

    Google Scholar 

  32. Maninis, K.K., Pont-Tuset, J., Arbeláez, P., Van Gool, L.: Convolutional oriented boundaries: from image segmentation to high-level tasks. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 819–833 (2017)

    Article  Google Scholar 

  33. Matula, P., Maška, M., Sorokin, D.V., Matula, P., Ortiz-de Solórzano, C., Kozubek, M.: Cell tracking accuracy measurement based on comparison of acyclic oriented graphs. PLoS ONE 10(12), e0144959 (2015)

    Article  Google Scholar 

  34. Meyer, F.: Morphological segmentation revisited. In: Space, Structure and Randomness: Contributions in Honor of Georges Matheron in the Field of Geostatistics, Random Sets and Mathematical Morphology, pp. 315–347 (2005)

    Google Scholar 

  35. Meyer, F., Oliveras Vergés, A., Salembier Clairon, P.J., Vachier, C.: Morphological tools for segmentation: connected operators and watersheds. Annales des télecommunications. Ann. Telecommun. 52(7–8), 366–379 (1997)

    Google Scholar 

  36. Murray, J.I., et al.: Automated analysis of embryonic gene expression with cellular resolution in c. elegans. Nat. Methods 5(8), 703–709 (2008)

    Google Scholar 

  37. Najman, L.: On the equivalence between hierarchical segmentations and ultrametric watersheds. J. Math. Imaging Vision 40(3), 231–247 (2011)

    Article  MathSciNet  Google Scholar 

  38. Najman, L., Cousty, J., Perret, B.: Playing with kruskal: algorithms for morphological trees in edge-weighted graphs. In: Hendriks, C.L.L., Borgefors, G., Strand, R. (eds.) ISMM 2013. LNCS, vol. 7883, pp. 135–146. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38294-9_12

    Chapter  Google Scholar 

  39. Najman, L., Schmitt, M.: Geodesic saliency of watershed contours and hierarchical segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 18(12), 1163–1173 (1996)

    Article  Google Scholar 

  40. Neumann, B., et al.: Phenotypic profiling of the human genome by time-lapse microscopy reveals cell division genes. Nature 464(7289), 721–727 (2010)

    Article  Google Scholar 

  41. Nishimura, K., Hayashida, J., Wang, C., Ker, D.F.E., Bise, R.: Weakly-supervised cell tracking via backward-and-forward propagation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12357, pp. 104–121. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58610-2_7

    Chapter  Google Scholar 

  42. Nunez-Iglesias, J., Kennedy, R., Parag, T., Shi, J., Chklovskii, D.B.: Machine learning of hierarchical clustering to segment 2d and 3d images. PLoS ONE 8(8), e71715 (2013)

    Article  Google Scholar 

  43. Ouyang, W., et al.: Bioimage model zoo: a community-driven resource for accessible deep learning in bioimage analysis. bioRxiv (2022)

    Google Scholar 

  44. Pape, C., Beier, T., Li, P., Jain, V., Bock, D.D., Kreshuk, A.: Solving large multicut problems for connectomics via domain decomposition. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1–10 (2017)

    Google Scholar 

  45. Payer, C., Štern, D., Neff, T., Bischof, H., Urschler, M.: Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 3–11. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_1

    Chapter  Google Scholar 

  46. Perret, B., Chierchia, G., Cousty, J., Guimaraes, S.J.F., Kenmochi, Y., Najman, L.: Higra: hierarchical graph analysis. SoftwareX 10, 100335 (2019)

    Article  Google Scholar 

  47. Perret, B., Cousty, J., Guimaraes, S.J.F., Maia, D.S.: Evaluation of hierarchical watersheds. IEEE Trans. Image Process. 27(4), 1676–1688 (2017)

    Article  MathSciNet  Google Scholar 

  48. Perret, B., Cousty, J., Guimarães, S.J.F., Kenmochi, Y., Najman, L.: Removing non-significant regions in hierarchical clustering and segmentation. Pattern Recogn. Lett. 128, 433–439 (2019)

    Article  Google Scholar 

  49. Pont-Tuset, J., Arbelaez, P., Barron, J.T., Marques, F., Malik, J.: Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Trans. Pattern Anal. Mach. Intell. 39(1), 128–140 (2016)

    Article  Google Scholar 

  50. Rempfler, M., et al.: Efficient algorithms for moral lineage tracing. In: IEEE International Conference on Computer Vision, pp. 4695–4704 (2017)

    Google Scholar 

  51. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  52. Schiegg, M., Hanslovsky, P., Kausler, B.X., Hufnagel, L., Hamprecht, F.A.: Conservation tracking. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2928–2935 (2013)

    Google Scholar 

  53. Schmidt, U., Weigert, M., Broaddus, C., Myers, G.: Cell detection with star-convex polygons. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 265–273. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_30

    Chapter  Google Scholar 

  54. Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 888–905 (2000)

    Article  Google Scholar 

  55. Stegmaier, J., et al.: Fast segmentation of stained nuclei in terabyte-scale, time resolved 3d microscopy image stacks. PLoS ONE 9(2), e90036 (2014)

    Article  Google Scholar 

  56. Stelzer, E.H., Strobl, F., Chang, B.J., Preusser, F., Preibisch, S., McDole, K., Fiolka, R.: Light sheet fluorescence microscopy. Nat. Rev. Methods Primers 1(1), 73 (2021)

    Article  Google Scholar 

  57. Stringer, C., Wang, T., Michaelos, M., Pachitariu, M.: Cellpose: a generalist algorithm for cellular segmentation. Nat. Methods 18(1), 100–106 (2021)

    Article  Google Scholar 

  58. Sugawara, K., Çevrim, Ç., Averof, M.: Tracking cell lineages in 3d by incremental deep learning. eLife 11, e69380 (2022)

    Google Scholar 

  59. Tan, M., Le, Q.: Efficientnet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114 (2019)

    Google Scholar 

  60. Tarjan, R.E.: Efficiency of a good but not linear set union algorithm. J. ACM (JACM) 22(2), 215–225 (1975)

    Article  MathSciNet  Google Scholar 

  61. Türetken, E., et al.: Network flow integer programming to track elliptical cells in time-lapse sequences. IEEE Trans. Med. Imaging 36(4), 942–951 (2016)

    Article  Google Scholar 

  62. Ulman, V., et al.: An objective comparison of cell-tracking algorithms. Nat. Methods 14(12), 1141–1152 (2017)

    Article  Google Scholar 

  63. Virtanen, P., et al.: Scipy 1.0: fundamental algorithms for scientific computing in python. Nat. Methods 17(3), 261–272 (2020)

    Google Scholar 

  64. Van der Walt, S., et al.: scikit-image: image processing in python. PeerJ 2, e453 (2014)

    Article  Google Scholar 

  65. Wolny, A., et al.: Accurate and versatile 3d segmentation of plant tissues at cellular resolution. eLife 9, e57613 (2020)

    Google Scholar 

  66. Xie, S., Tu, Z.: Holistically-nested edge detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1395–1403 (2015)

    Google Scholar 

  67. Yang, B., et al.: Daxi–high-resolution, large imaging volume and multi-view single-objective light-sheet microscopy. Nat. Methods 19(4), 461–469 (2022)

    Article  Google Scholar 

  68. Yarkony, J., Ihler, A., Fowlkes, C.C.: Fast planar correlation clustering for image segmentation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 568–581. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33783-3_41

    Chapter  Google Scholar 

  69. Yoo, A.B., Jette, M.A., Grondona, M.: SLURM: simple linux utility for resource management. In: Feitelson, D., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2003. LNCS, vol. 2862, pp. 44–60. Springer, Heidelberg (2003). https://doi.org/10.1007/10968987_3

    Chapter  Google Scholar 

  70. Zimmer, C., Labruyere, E., Meas-Yedid, V., Guillén, N., Olivo-Marin, J.C.: Segmentation and tracking of migrating cells in videomicroscopy with parametric active contours: a tool for cell-based drug testing. IEEE Trans. Med. Imaging 21(10), 1212–1221 (2002)

    Article  Google Scholar 

Download references

Acknowledgements

We thank Ahmed Abbas, Ahmet Can Solak, Alexandre Falcão, Guillaume Le Treut, Ilan Theodoro, Laurent Najman, Paul Swoboda, Sandy Schmid, and Teun Huijben for the discussions regarding this work and Chan Zuckerberg Biohub for the support.

Author information

Authors and Affiliations

  1. Chan Zuckerberg Biohub, San Francisco, USA

    Jordão Bragantini, Merlin Lange & Loïc Royer

Authors
  1. Jordão Bragantini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Merlin Lange
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Loïc Royer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jordão Bragantini .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (zip 68283 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bragantini, J., Lange, M., Royer, L. (2025). Large-Scale Multi-hypotheses Cell Tracking Using Ultrametric Contours Maps. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15137. Springer, Cham. https://doi.org/10.1007/978-3-031-72986-7_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72986-7_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72985-0

  • Online ISBN: 978-3-031-72986-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics