net.imglib2.algorithm.metrics.segmentation

Class MultiMetrics

java.lang.Object

net.imglib2.algorithm.metrics.segmentation.MultiMetrics

public class MultiMetrics
extends Object

Compute a set of metrics at a specific threshold between the labels of a predicted and of a ground-truth image. In the context of instance segmentation, the labels are the pixel values. The metrics score are obtained by matching ground-truth with prediction labels using the Munkres-Kuhn algorithm. The assignment is based on a cost matrix, in which each element is the negation of the IoU (a.k.a Jaccard index) between ground-truth and prediction labels if the IoU is greater than or equal to the threshold and 0 otherwise. The metrics are the following:

TP = number of matched ground-truth labels

FP = number of unmatched predicted labels

FN = number of unmatched ground-truth labels

Precision = TP / (TP + FP)

Recall = TP / (TP + FN)

F1 = 2 * Precision * Recall / (Precision + Recall)

Accuracy = TP / ( TP + FP + FN)

Mean matched IoU = sum of the matched labels IoU / TP

Mean true IoU = sum of the matched labels IoU / number of ground-truth labels

Pixels with value 0 are considered background and are ignored during the metrics calculation. If the ground-truth image is only background (no labels), then the metrics returns TP=FP=FN=0 and NaN for the others.

The metrics expect images of dimensions XYZT, where Z and T can be of depth 1. The metrics scores are calculated for each ground-truth label in each XYZ volume (XY if dimension Z is of depth 1), and averaged over the total number of ground-truth labels in the XYZT volume. If the Z dimension has depth greater than 1, then the labels are considered 3D and pixels of equal values at different depths are considered to be part of the same labeling.

Finally, if the image stack does not fit in memory, you can use LazyMultiMetrics to compute running metrics scores for which images can be added one at time.

Note: the accuracy is in some cases also called the average precision (see DSB challenge).

This class was inspired from the StarDist repository.

Author:

Joran Deschamps

See Also:

StarDist metrics

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MultiMetrics.Metrics Metrics computed by MultiMetrics objects.
protected static class	MultiMetrics.MetricsSummary An object holding the sum of TP, FP, FN and IoU values, accumulated over the T dimension.

Constructor Summary

Constructors

Constructor and Description
MultiMetrics()

Method Summary

Modifier and Type	Method and Description
protected static double[][]	computeCostMatrix(ConfusionMatrix cM, double threshold) Computes a cost matrix using the confusion matrix cM.
protected static MultiMetrics.MetricsSummary	computeFinalScores(ConfusionMatrix confusionMatrix, double[][] costMatrix, double threshold) Compute all metrics scores by matching ground-truth labels (row indices) with predicted labels (column indices).
static <T,I extends net.imglib2.type.numeric.IntegerType<I>,U,J extends net.imglib2.type.numeric.IntegerType<J>> HashMap<MultiMetrics.Metrics,Double>	computeMetrics(net.imglib2.roi.labeling.ImgLabeling<T,I> groundTruth, net.imglib2.roi.labeling.ImgLabeling<U,J> prediction, double threshold) Compute the global metrics scores between labels from a ground-truth and a predicted image.
static <I extends net.imglib2.type.numeric.IntegerType<I>,J extends net.imglib2.type.numeric.IntegerType<J>> HashMap<MultiMetrics.Metrics,Double>	computeMetrics(net.imglib2.RandomAccessibleInterval<I> groundTruth, net.imglib2.RandomAccessibleInterval<J> prediction, double threshold) Compute the global metrics scores between labels from a ground-truth and a predicted image.
protected static double	getLocalIoUScore(ConfusionMatrix cM, int iGT, int jPred, double threshold) Compute the IoU between the ground-truth label iGT and the prediction label jPred.
protected static <I extends net.imglib2.type.numeric.IntegerType<I>,J extends net.imglib2.type.numeric.IntegerType<J>> MultiMetrics.MetricsSummary	runAverageOverTime(net.imglib2.RandomAccessibleInterval<I> groundTruth, net.imglib2.RandomAccessibleInterval<J> prediction, double threshold) Compute the metrics score per time-point and return the aggregated scores.
protected static <I extends net.imglib2.type.numeric.IntegerType<I>,J extends net.imglib2.type.numeric.IntegerType<J>> MultiMetrics.MetricsSummary	runSingle(net.imglib2.RandomAccessibleInterval<I> groundTruth, net.imglib2.RandomAccessibleInterval<J> prediction, double threshold) Compute the metrics scores between ground-truth and prediction images.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultiMetrics

public MultiMetrics()

Method Detail

computeMetrics

public static <T,I extends net.imglib2.type.numeric.IntegerType<I>,U,J extends net.imglib2.type.numeric.IntegerType<J>> HashMap<MultiMetrics.Metrics,Double> computeMetrics(net.imglib2.roi.labeling.ImgLabeling<T,I> groundTruth,
                                                                                                                                                                            net.imglib2.roi.labeling.ImgLabeling<U,J> prediction,
                                                                                                                                                                            double threshold)

Compute the global metrics scores between labels from a ground-truth and a predicted image. The method expects images of dimension XYZT. The scores are computed over each XYZ volume (or XY if Z is of depth 1) and accumulated. The mean matched IoU and mean true IoU scores are then averaged over all true positives or all ground-truth labels, respectively. If both images are empty (only pixels with value 0), then the metrics score is NaN.

The threshold is the minimum IoU between a ground-truth and a prediction label at which two labels are considered a potential match.

This method is not compatible with ImgLabeling with intersecting labels.

Type Parameters:

T - Label type associated to the ground-truth

I - Ground-truth pixel type

U - Label type associated to the prediction

J - Prediction pixel type

Parameters:

groundTruth - Ground-truth image

prediction - Predicted image

threshold - Threshold

Returns:

Metrics score

computeMetrics

public static <I extends net.imglib2.type.numeric.IntegerType<I>,J extends net.imglib2.type.numeric.IntegerType<J>> HashMap<MultiMetrics.Metrics,Double> computeMetrics(net.imglib2.RandomAccessibleInterval<I> groundTruth,
                                                                                                                                                                        net.imglib2.RandomAccessibleInterval<J> prediction,
                                                                                                                                                                        double threshold)

Compute the global metrics scores between labels from a ground-truth and a predicted image. The method expects images of dimension XYZT. The scores are computed over each XYZ volume (or XY if Z is of depth 1) and accumulated. The mean matched IoU and mean true IoU scores are then averaged over all true positives or all ground-truth labels, respectively. If both images are empty (only pixels with value 0), then the metrics score is NaN.

The threshold is the minimum IoU between a ground-truth and a prediction label at which two labels are considered a potential match.

Type Parameters:

I - Ground-truth pixel type

J - Prediction pixel type

Parameters:

groundTruth - Ground-truth image

prediction - Predicted image

threshold - Threshold

Returns:

Metrics scores

runAverageOverTime

protected static <I extends net.imglib2.type.numeric.IntegerType<I>,J extends net.imglib2.type.numeric.IntegerType<J>> MultiMetrics.MetricsSummary runAverageOverTime(net.imglib2.RandomAccessibleInterval<I> groundTruth,
                                                                                                                                                                      net.imglib2.RandomAccessibleInterval<J> prediction,
                                                                                                                                                                      double threshold)

Compute the metrics score per time-point and return the aggregated scores. The method expect XYZT images, where Z can be of dimension 1, and run the metrics computation on each XYZ volume.

Type Parameters:

I - Ground-truth pixel type

J - Prediction pixel type

Parameters:

groundTruth - Ground-truth image

prediction - Predicted image

threshold - Threshold

Returns:

Metrics summary

runSingle

protected static <I extends net.imglib2.type.numeric.IntegerType<I>,J extends net.imglib2.type.numeric.IntegerType<J>> MultiMetrics.MetricsSummary runSingle(net.imglib2.RandomAccessibleInterval<I> groundTruth,
                                                                                                                                                             net.imglib2.RandomAccessibleInterval<J> prediction,
                                                                                                                                                             double threshold)

Compute the metrics scores between ground-truth and prediction images.

Type Parameters:

I - Ground-truth pixel type

J - Prediction pixel type

Parameters:

groundTruth - Ground-truth image

prediction - Predicted image

threshold - Threshold

Returns:

Metrics summary

computeCostMatrix

protected static double[][] computeCostMatrix(ConfusionMatrix cM,
                                              double threshold)

Computes a cost matrix using the confusion matrix cM. Each element cij of the cost matrix is the negation of the IoU between ground-truth label i and prediction label j. If the IoU is smaller than the threshold, then cij is set to 0. The negation allows formulating the problem of matching labels as a minimum cost assignment.

The cost matrix is always rectangular, with empty rows added to ensure that the number of prediction label is strictly superior to the number of ground-truth labels. This is necessary to ensure that the Munkres-Kuhn algorithm returns coherent results at low ground-truth label numbers.

Parameters:

cM - Confusion matrix

threshold - Threshold

Returns:

Cost matrix

getLocalIoUScore

protected static double getLocalIoUScore(ConfusionMatrix cM,
                                         int iGT,
                                         int jPred,
                                         double threshold)

Compute the IoU between the ground-truth label iGT and the prediction label jPred. If the IoU is smaller than the threshold, the method returns 0.

Parameters:

cM - Confusion matrix

iGT - Index of the ground-truth label

jPred - Index of the prediction label

threshold - Threshold

Returns:

IoU between label iGT and jPred, or 0 if the IoU is smaller than the threshold

computeFinalScores

protected static MultiMetrics.MetricsSummary computeFinalScores(ConfusionMatrix confusionMatrix,
                                                                double[][] costMatrix,
                                                                double threshold)

Compute all metrics scores by matching ground-truth labels (row indices) with predicted labels (column indices). The assignment is performed by the Munkres-Kuhn algorithm using the costMatrix. All metrics scores can be accessed by calling MultiMetrics.MetricsSummary getScores() after this method.

Parameters:

confusionMatrix - Confusion matrix

costMatrix - Cost matrix

threshold - Threshold

Returns:

Metrics summary