Twofold consensus for boundary detection ground truth
Title: Twofold consensus for boundary detection ground truth
Journal: Knowledge-Based Systems
Abstract: In the evaluation of boundary detection methods it is common to use as ground truth a set of boundary images that are hand-made by human experts. This work proposes a novel representation of this ground truth. More specifically, we propose to combine the hand-made boundary images into a set-based consensus, which is constructed from the concordances and discordances among the images. We study the theoretical and visual properties of this consensus and present an application to boundary image quality evaluation.
Keywords: Boundary detection; Quality evaluation; Binary image; Consensus image; Ground truth.
Cite as: C. Lopez-Molina, B. De Baets, H. Bustince, Twofold consensus for boundary detection ground truth, Knowledge-Based Systems, Volume 98, 15 April 2016, Pages 162-171.
One of the obstacles we identify in the process of evaluating the quality of a boundary image is the inherent difficulty in obtaining and representing ground truth solutions. Differently from what happens in other image processing tasks, it is unclear how to obtain perfect solutions the automatically generated boundary images can be compared to. Therefore, in practical terms, the boundary detection task is redefined as marking up the boundaries a human would consider to be worth tagging in an image. This enforces the ground truth to be human-made boundary images. Although this is not a problem itself, it leads to multiple situations for which no answer has been provided in the literature, the most relevant being that in which different humans produce very divergent solutions. The discrepancies can result from marking up (or not) certain objects whose importance in the image is debatable, but also from locating their boundaries at different positions.
In this work we analyze the management of multiple ground truth. Specifically, we consider the generation of a singular representation of the boundary images generated by different humans. Our proposal consist of creating the so-called Twofold-consensus ground truth (TCGT), which is given as a complementary representation of (a) the boundaries in which all humans agree (strong consensus) and (b) the boundaries tagged by at least one human (weak consensus).
Our proposal is specified and studied. Then, we propose a quantitative error measure for boundary detection based on the TCGT.
Code (in the KITT): The following pieces of code are of interest for the study and/or use of the developments in this work:
- Function TCGT.m (package boundaryImageComparison)
- The paper at Elsevier Editorial System;
Related works (in the KITT):
- Separability criteria for the evaluation of boundary detection benchmarks;
- Quantitative error measures for edge detection.
Related works (web):
- Automatic generation of consensus ground truth for the comparison of edge detection techniques (at Science Direct).