Evaluation Methodology

The goal of both challenges is to detect, segment, and label all cells in each video. Cells entering the field of view should be detected, segmented, and labeled as well. Whereas cells dividing mitotically should be identified and the daughter cells labeled differently, cells splitting into several parts, clearly due to fragmentation of elongated cell membranes, should keep the same label while being separated.

 

The detection accuracy (DET) of the methods, understood as how accurately each given object has been identified, is based on comparison of the nodes of acyclic oriented graphs representing objects in both the GT and each tested method. Numerically, DET is defined as a normalized Acyclic Oriented Graph Matching (AOGM-D) measure for detection:

DET = 1- min(AOGM-D, AOGM-D0)/AOGM-D0

where AOGM-D is the cost of transforming a set of nodes provided by the participant into the set of GT nodes; AOGM-D0 is the cost of creating the set of GT nodes from scratch (i.e., it is AOGM-D for empty detection results). The minimum operator in the numerator prevents from having a final negative value when it is cheaper to create the reference set of nodes from scratch than to transform the computed set of nodes into the reference one. The normalization ensures that DET always falls in the [0,1] interval, with higher values corresponding to better detection performance.

The segmentation accuracy (SEG) of the methods, understood as how well the segmented regions of the cells match the actual cell or nucleus boundaries, is measured by comparing the segmented objects with the ground truth (GT) consisting of the reference annotation of selected frames (2D) and/or image planes (in the 3D cases). Numerically, the SEG measure is based on the Jaccard similarity index, with a detailed description given in SEG.pdf.

The tracking accuracy (TRA) of the methods understood as how accurately each given object has been identified and followed in successive frames is based on comparison of acyclic oriented graphs representing the time development of objects in both the GT and each tested method. Numerically, TRA is defined as a normalized Acyclic Oriented Graph Matching (AOGM) measure:

TRA = 1- min(AOGM, AOGM0)/AOGM0

where AOGM0 is the AOGM value required for creating the reference graph from scratch (i.e., it is the AOGM value for empty tracking results). The minimum operator in the numerator prevents from having a final negative value when it is cheaper to create the reference graph from scratch than to transform the computed graph into the reference graph. The normalization ensures that TRA always falls in the [0,1] interval, with higher values corresponding to better tracking performance.

 

Command-line software packages that implement the DET, SEG and TRA measures are made publicly available, along with the instructions required to run the packages, These packages are used for the official evaluation of the algorithms by the challenge organizers and can be used by the participants to evaluate and tune their algorithms too.

 

The ranking of the methods provided in the Cell Segmentation Benchmark is based on the average of DET and SEG measures to allow a direct comparison of the methods:

OPCSB = 0.5⋅(DET + SEG).

The ranking of the methods provided in the Cell Tracking Benchmark is based on the average of SEG and TRA measures to allow a direct comparison of the methods:

OPCTB = 0.5⋅(SEG + TRA).