Measures for ranking cell trackers without manual validation
Introduction
Automated cell tracking has become an important tool in a wide range of biological studies, including anti-cancer drug screening, measuring the proliferation of immune system cells, and conducting wound healing assays [1], [2]. Systems for tracking cells often comprise two separate steps: cell detection and association of detected cells across frames [2], [3]. In this paper, we focus on the association step, and by cell tracker we mean an algorithm for data association.
Many cell trackers have parameters that need to be specified, for example, the process noise covariance for the Kalman filter used in the method of Li et al. [4], and the weights for association costs in the algorithm by Padfield et al. [3]. Given a cell video processed by a cell detector, users naturally want to select a tracking algorithm with appropriate parameter values that leads to the most accurate reconstruction of cell tracks. We refer to the accuracy of track reconstruction as tracking performance.
Existing measures of tracking performance [5], [6], [4] require the availability of ground truth (GT) information, for example, identities of cells throughout the video. A common approach is to generate a GT by manually annotating a subset of video frames. However, the best choice of tracking algorithm and its associated parameter settings can vary between videos, and it can be very time consuming to manually annotate every new video. Therefore, an important and open research problem in automated cell tracking is how to select algorithms and their associated parameter settings without access to the GT for videos.
To address this problem, our aim is to develop a measure for ranking trackers according to their performance without the need for a GT. The challenge in developing such measures is the absence of labeled training data. We propose a method for estimating the performance of a tracker based solely on the information available from the tracking results, such as the lengths of links made by a tracker. Our method relies on an assumption that the distribution of the lengths of wrong associations can be estimated from the sample of distances between locations within frames (Section 4.3.1). Our evaluation on real and synthetic videos shows that the assumption holds in practical scenarios. The main application for our measures is selecting the most accurate tracking algorithm (and its parameters) for a given video and a fixed detection step. This problem can arise, for example, when cells of the same type are recorded under different treatments, so that the visual appearance of the cells is the same, but the motility varies across videos.
Although, in this paper, we consider cell tracking as our main application, our method is sufficiently general to be applicable to other domains, such as particle tracking or vehicle tracking. This is due to the fact that we develop our formulations upon a general points association problem. Furthermore, we approach the tracking problem from the perspective of pattern recognition by breaking tracks down into inter-frame links, and categorizing the links into positive and negative classes. The tracking problem then essentially becomes recognizing positive links from the set of all possible links on a given detection. This allows us to apply concepts of precision and recall in our solution.
In summary, the contributions of our paper are as follows: (a) we present several novel measures for ranking cell trackers according to their performance. These measures do not require a GT (Section 4); and (b) we evaluate our proposed measures using both real and synthetic videos for different trackers, and show that our measures correlate with tracking performance in practical cell tracking scenarios (Section 5).
Section snippets
Related work
Over recent decades, there have been many proposed cell tracking methods [7], [8]. Each of these methods has a number of parameter values that need to be specified by the user. Rittscher (2010) remarks: “it is often not clear which particular tracking algorithm is well suited for the given data type. It would be helpful if the decision on what type of algorithm should be used, or what particular parameter setting should be used, could be made automatically” [8]. We address this challenge by
Cell tracking preliminaries
Consider a video that has been processed by a cell detector (e.g., presented in [20]). The cell detector identifies cells in each frame of the video. A location is a k-dimensional vector , where , i=1,…,k. For example, the location can be a vector comprising the centroid position, fluorescence and size of a cell.1
Measuring performance without ground truth
Our approach for solving the problem stated in the previous section is to develop a measure that estimates the tracking performance from the information available as a result of tracking. For example, one can know the number of links made by the tracker in every frame. Furthermore, one can know the lengths of the links made by the tracker. Our baseline measure presented in the next section uses the number of links a tracker makes between consecutive frames.
Evaluation
The aims of our evaluation have been to (i) validate the proposed measures in practical scenarios, and (ii) study the reliability of the measures under conditions that are expected to hamper our method. In order to achieve these aims, we use different cell trackers previously proposed in the literature, and vary the parameter values used in these trackers. Furthermore, we use a range of real and synthetic videos as inputs.5
Discussion
The advent of high content assays in screening applications makes it impracticable to tune a cell tracker for every new video. Consider a number of videos produced in a course of a biological experiment. In the experiment, cells of the same type are recorded under different treatment conditions, and the treatment is expected to affect cell motility or lifetimes. A visual inspection of a few frames from one of the videos can guide the choice of a cell detection algorithm and its parameters. The
Conclusions
In this paper, we address the problem of ranking cell trackers according to their performance. By cell tracker we mean a point association algorithm, and by performance we mean the accuracy of track reconstruction. We have developed several novel measures for estimating the tracking performance without the need for the ground truth. Our measures can be used to automatically select the most appropriate tracker and its parameters for a given set of cell detection results. One of our measures
Conflict of interest statement
None declared.
Acknowledgments
We would like to thank Dr. Khuloud Jaqaman (Harvard Medical School) for the advice regarding u-track; Dr. Zhaozheng Yin (Carnegie Mellon University) for sharing the videos of wound healing assays; and Dr. Daniel Day (Swinburne University of Technology) for supplying the microgrids. This work is partially supported by National ICT Australia (NICTA). NICTA is funded by the Australian Government's Backing Australia's Ability initiative, in part through the Australian Research Council.
Andrey Kan received BS degree in 2003 in Tashkent and MS degree in 2005 in Moscow. His research interests include computer vision and data mining, especially in biological applications.
References (23)
- et al.
Cell population tracking and lineage construction with spatiotemporal context
Medical Image Analysis
(2008) - et al.
Tracking in cell and developmental biology
Seminars in Cell and Developmental Biology
(2009) - et al.
Scattering of exciting light by live cells in fluorescence confocal imagingphototoxic effects and relevance for FRAP studies
Biophysical Journal
(2007) - et al.
Robust visual tracking with structured sparse representation appearance model
Pattern Recognition
(2012) - et al.
Real-time tracking of multiple objects in space-variant vision based on magnocellular visual pathway
Pattern Recognition
(2002) - et al.
Hand tracking in a natural conversational environment by the interacting multiple model and probabilistic data association (IMM-PDA) algorithm
Pattern Recognition
(2005) - et al.
A shape-based voting algorithm for pedestrian detection and tracking
Pattern Recognition
(2011) - et al.
A single-cell pedigree analysis of alternative stochastic lymphocyte fates
Proceedings of the National Academy of Sciences of the United States of America
(2009) - T. Kanade, Z. Yin, R. Bise, S. Huh, S.E. Eom, M. Sandbothe, M. Chen, Cell image analysis: algorithms, system and...
- et al.
Coupled minimum-cost flow cell tracking for high-throughput quantitative analysis
Medical Image Analysis
(2010)
An automatic system for in vitro cell migration studies
Journal of Microscopy
Cited by (7)
Automatic 3D tracking system for large swarm of moving objects
2016, Pattern RecognitionCitation Excerpt :A system for tracking multiple players in sports videos was proposed in [13]. A data association strategy for cell tracking was proposed in [14]. The mutual interaction among multiple humans was modeled to guide tracking in [15].
Ranking cell tracking systems without manual validation
2015, Pattern Recognition LettersCitation Excerpt :Moreover, the accuracy of cell boundaries is categorized by a user into “accepted” leading to a correct detection and “not accepted” leading to a missing detection. Finally, the F-score correlates with some other common performance measures [7]. In summary, we propose a novel method for ranking cell tracking systems with minimum input (estimate of the cell size) required for each new video.
An Automated Partial Derivative Based Method for Detecting and Monitoring Moving Objects
2023, Journal of Machine and ComputingMachine learning applications in cell image analysis
2017, Immunology and Cell BiologyUncertainty Footprint: Visualization of Nonuniform Behavior of Iterative Algorithms Applied to 4D Cell Tracking
2017, Computer Graphics Forum
Andrey Kan received BS degree in 2003 in Tashkent and MS degree in 2005 in Moscow. His research interests include computer vision and data mining, especially in biological applications.
Christopher Leckie is a Professor at the Department of Computing and Information Systems at the University of Melbourne in Australia. He has made significant theoretical and practical contributions in efficiently analysing large volumes of data in resource-constrained environments, such as wireless sensor networks.
James Bailey is an Associate Professor at the Department of Computing and Information Systems at the University of Melbourne. He received his PhD from the University of Melbourne in 1998 and BSc and BE from the same University in 1993 and 1994. His research interests are in data mining and bioinformatics.
John Markham received a BEng (Electronic) from the Swinburne Institute of Technology; after working in the electronic publishing industry, he returned to the University of Melbourne to complete a BSc (Hons) majoring in physics. He then completed a PhD at Melbourne in an area of theoretical computational physics called lattice gauge theory.
Rajib Chakravorty is a Senior Research Engineer at the National ICT Australia (NICTA). His research looks at converting the huge amount of microscopic image and video data, generated by biological research labs around the world, into meaningful information. He is looking at identifying and developing a platform which will intelligently cope with variability of bio-imaging data.