DRIVE: Results

Introduction

So far, a total of eight vessel segmentation methods have been tested on the DRIVE database. Each method is referenced on this page by a number and the name of the first author of the paper in which the method was presented. At the end of this page, references are provided for further information.

All results pertain to the test set of the DRIVE database, that consists of 20 images that have been hand-labelled twice. One of these labellings is used as gold standard. The other labelling is included below and is referred to as the human observer method. We also show, for reference, the performance of a trivial method that assigns all pixels to the most likely class, namely the background. The remaining methods are produced by dedicated computer algorithms. They are all completely automatic, no manual initializations or interactions are needed. In the future, interactive methods may be added, together with information about the required segmentation time.

Performance measurements

Performance is given as accuracy (part of pixels correctly classified) and kappa values (a measure for observer agreement, where the two observers are the gold standard and the segmentation method. There are two classes (vessel and background), and only pixels inside the field of view are taken into account (the mask images supplied in the DRIVE database provide the field of view for each image).

Two types of results are supported: soft and hard classification. Soft classification results are used to produce ROC curves (sensitivity versus 1 - specificity), and from these curves Az, the area under the curve. is computed. We produce the ROC curves simply by varying the threshold on the soft classification image and not by fitting a model to the points.

Hard classification results can be produced from the soft classification results by thresholding at a certain value. Contributors of methods that procude soft classifications are advised to select an optimal threshold (that is, one that maximizes accuracy) from the results of their method on the training cases, and use this threshold to produce the hard classification. This procedure was used for all soft classification methods available so far.

Results

The table below shows an overview of the performance of the different methods. The rightmost column shows the name of the person who implemented the algorithm.

Method

Accuracy

Kappa

Az

Implementation

Human observer

0.9473 (0.0048)

0.7589

n/a

-

Staal [1]

0.9442 (0.0065)

0.7345

0.9520

J. Staal

Niemeijer [2]

0.9416 (0.0065)

0.7145

0.9294

M. Niemeijer

Zana [3]

0.9377 (0.0077)

0.6971

0.8984

M. Niemeijer

Al-Diri [7]

0.9258 (0.0126)

0.6716

n/a

B. Al-Diri

Jiang [4]

0.9212 (0.0076)

0.6399

0.9114

J. Staal

Martínez-Pérez [5]

0.9181 (0.0240)

0.6389

n/a

M. Niemeijer

Chaudhuri [6]

0.8773 (0.0232)

0.3357

0.7878

M. Niemeijer

All background

0.8727 (0.0123)

0

n/a

-

The figure below shows the ROC curves of the different methods that have been tested on the DRIVE database. Methods which result in a binary segmentation are shown as single points in the ROC-plot. To get results on individual images, use the result browser.



References

[1] J.J. Staal, M.D. Abramoff, M. Niemeijer, M.A. Viergever, B. van Ginneken, "Ridge based vessel segmentation in color images of the retina", IEEE Transactions on Medical Imaging, 2004, vol. 23, pp. 501-509.

[2] M. Niemeijer, J.J. Staal, B. van Ginneken, M. Loog, M.D. Abramoff, "Comparative study of retinal vessel segmentation methods on a new publicly available database", in: SPIE Medical Imaging, Editor(s): J. Michael Fitzpatrick, M. Sonka, SPIE, 2004, vol. 5370, pp. 648-656.

[3] F. Zana and J. Klein, Segmentation of vessel-like patterns using mathematical morphology and curvature evaluation, IEEE Transactions on Image Processing 10(7), pp. 1010-1019, 2001.

[4] X. Jiang and D. Mojon, Adaptive local thresholding by verification-based multithreshold probing with application to vessel detection in retinal images, IEEE Transactions on Pattern Analysis and Machine Intelligence 25(1), pp. 131-137, 2003.

[5] M. Martínez-Pérez, A. Hughes, A. Stanton, S. Thom, A. Bharath, and K. Parker, Scale-space analysis for the characterisation of retinal blood vessels, in Medical Image Computing and Computer-Assisted Intervention - MICCAI’99, C. Taylor and A. Colchester, eds., pp. 90-97, 1999.

[6] S. Chaudhuri, S. Chatterjee, N. Katz, M. Nelson, and M. Goldbaum, Detection of blood vessels in retinal images using two-dimensional matched filters, IEEE Transactions on Medical Imaging 8(3), pp. 263-269, 1989.

[7] B. Al-Diri, A. Hunter, D. Steel, An Active Contour Model for Segmenting and Measuring Retinal Vessels, IEEE Transactions on Medical Imaging, 28(9), pp. 1488-97, 2009.