History of the Cell Tracking Challenge
The Cell Tracking Challenge was launched in 2012, with the aim of fostering the development of novel, robust cell segmentation and tracking algorithms, and helping the developers with the evaluation of their new algorithmic developments. Over its more than a decade long existence, six fixed-deadline challenge editions have been organized, and since February 2017, the challenge is open for online submissions that are monthly evaluated, ranked, and posted on the challenge website.
In the first edition, organized under the auspices of ISBI 2013 in San Francisco (CA, USA), we called for segmentation and tracking algorithms capable of analyzing time-lapse sequences of fluorescently labeled cells and nuclei moving in 2D and 3D environments. A report covering the logistics, methods, and results of the challenge was published in Bioinformatics [1]. A thorough evaluation of the newly developed tracking performance measure used in this and future editions of the challenge was published in PLoS One [2].
Given the success of the first edition of the challenge, the second edition was organized under the auspices of ISBI 2014 in Beijing (China). To broaden the scope of the challenge and to increase the interest from potential participants, the dataset repository was extended by 3D+t developmental fluorescence microscopy data, and 2D+t phase contrast and differential interference contrast microscopy data. We also introduced a new generation of the computer-generated datasets produced by our cell simulator [3].
In the third edition, held under the auspices of ISBI 2015 in Brooklyn (NY, USA), the challenge was consolidated by the increased number of participants and submissions, especially for the most challenging datasets. Given the growing relevance of high-throughput large-scale embryonic data, a new dataset consisting of Drosophila melanogaster embryonic data imaged using light-sheet microscopy was added.
A comprehensive description of the three editions of the challenge along with the results of 21 algorithms over the repository of 13 different datasets was published in Nature Methods in 2017 [4].
In October 2018, we introduced a new segmentation-centric benchmark as part of the fourth edition of the challenge held under the auspices of ISBI 2019 in Venice (Italy). Furthermore, the dataset repository was extended by adding 3D+time cartographic projections of Tribolium Castaneum embryos imaged using light-sheet microscopy and real and computer-generated time-lapse sequences of single cells with filopodial protrusions [5].
The fifth edition, organized as part of ISBI 2020, broadened the scope of the challenge by adding two brightfield microscopy datasets and one fully 3D+time dataset of developing Tribolium Castaneum embryos. Furthermore, silver reference segmentation annotations were released for the training videos of nine existing datasets to facilitate the tuning of competing methods. The submissions were evaluated, ranked, and announced at the virtual ISBI 2020 challenge workshop (see PowerPoint presentation [178 MB] and MP4 video [583 MB]).
In the sixth edition attached to ISBI 2021, we expanded silver reference segmentation annotations for the training videos of 13 existing datasets and focused on the development of methods that exhibit better generalizability and work across most, if not all, of the existing datasets, instead of being optimized for one or a few datasets only.
A report that comprehensively analyzes the results of 89 algorithms collected before June 2022 over the repository of 20 different datasets, and introduces the outcomes of three insightful studies about the relationship between the technical and biological performance of the benchmarked algorithms and the dataset and annotation properties, as well as about the generalizability and the reusability of top-performing algorithms, was published in Nature Methods in 2023 [6].
Challenge-Related Publications
[1] M. Maška, V. Ulman, D. Svoboda, P. Matula, P. Matula, C. Ederra, A. Urbiola, T. España, S. Venkatesan, D. M. W. Balak, P. Karas, T. Bolcková, M. Štreitová, C. Carthel, S. Coraluppi, N. Harder, K. Rohr, K. E. G. Magnusson, J. Jaldén, H. M. Blau, O. Dzyubachyk, P. Křížek, G. M. Hagen, D. Pastor-Escuredo, D. Jimenez-Carretero, M. J. Ledesma-Carbayo, A. Muñoz-Barrutia, E. Meijering, M. Kozubek, C. Ortiz-de-Solorzano. A benchmark for comparison of cell tracking algorithms. Bioinformatics, 2014.
[2] P. Matula, M. Maška, D. V. Sorokin, P. Matula, C. Ortiz-de-Solorzano, M. Kozubek, Cell tracking accuracy measurement based on comparison of acyclic oriented graphs. PLoS One, 2015.
[3] D. Svoboda, V. Ulman. MitoGen: A framework for generating 3D synthetic time-lapse sequences of cell populations in fluorescence microscopy. IEEE Transactions on Medical Imaging, 2017.
[4] V. Ulman, M. Maška, K. E. G. Magnusson, O. Ronneberger, C. Haubold, N. Harder, P. Matula, P. Matula, D. Svoboda, M. Radojevic, I. Smal, K. Rohr, J. Jaldén, H. M. Blau, O. Dzyubachyk, B. Lelieveldt, P. Xiao, Y. Li, S.-Y. Cho, A. C. Dufour, J. C. Olivo-Marin, C. C. Reyes-Aldasoro, J. A. Solis-Lemus, R. Bensch, T. Brox, J. Stegmaier, R. Mikut, S. Wolf, F. A. Hamprecht, T. Esteves, P. Quelhas, Ö. Demirel, L. Malmström, F. Jug, P. Tomancak, E. Meijering, A. Muñoz-Barrutia, M. Kozubek, C. Ortiz-de-Solorzano. An objective comparison of cell-tracking algorithms. Nature Methods, 2017.
[5] D. V. Sorokin, I. Peterlík, V. Ulman, D. Svoboda, T. Nečasová, K. Morgaenko, L. Eiselleová, L. Tesařová, M. Maška. FiloGen: A model-based generator of synthetic 3-D time-lapse sequences of single motile cells with growing and branching filopodia. IEEE Transactions on Medical Imaging, 2018.
[6] M. Maška, V. Ulman, P. Delgado-Rodriguez, E. Gómez-de-Mariscal, T. Nečasová, F. A. G. Peña, T. I. Ren, E. M. Meyerowitz, T. Scherr, K. Löffler, R. Mikut, T. Guo, Y. Wang, J. P. Allebach, R. Bao, N. M. Al-Shakarji, G. Rahmon, I. E. Toubal, K. Palaniappan, F. Lux, P. Matula, K. Sugawara, K. E. G. Magnusson, L. Aho, A. R. Cohen, A. Arbelle, T. Ben-Haim, T. R. Raviv, F. Isensee, P. F. Jäger, K. H. Maier-Hein, Y. Zhu, C. Ederra, A. Urbiola, E. Meijering, A. Cunha, A. Muñoz-Barrutia, M. Kozubek, C. Ortiz-de-Solórzano. The Cell Tracking Challenge: 10 years of objective benchmarking, Nature Methods, 2023.
Acknowledgment
We appreciate the support from the program for large research infrastructures of the Ministry of Education, Youth and Sports (MEYS) of the Czech Republic (Czech-BioImaging Projects LM2023050 and CZ.02.1.01/0.0/0.0/18_046/0016045) for providing state-of-the-art computational and storage resources. We also gratefully acknowledge the support of the NVIDIA Corporation and their donation of the Quadro P6000 GPU used for the earlier evaluation of challenge results, and thank the IT4Innovations National Supercomputing Center for hosting the computations of silver reference segmentation annotations on their cluster.