What criteria should be used to decide whether an automated detection overlaps with a reference annotation?

Question

For those who use automated detection and routinely need to evaluate detector performance, what constitutes an overlap between an automated detection and a reference annotation?

Most of my bioacoustics work is with terrestrial species (birds). I've written plenty of code for comparing whether automated detections are lining up with annotations in a "truth" data set. Depending on the motivating question, I've gone back and forth between coding it as, "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table", vs. "ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive". Depending on motivating question and vocalization type of interest, it seems this decision point is another place where error might be introduced. There are likely creative, robust ways to handle this that I'm overlooking. For those routinely tasked with comparing an automated detector's performance to a truth/reference annotation data set, how do you typically handle this?

Inspired by this relevant question: Software options for diagnosing the performance of a detection routine

Answer 1

In the R package ohun any overlapping detection is taken as a true positive, as it would be arbitrary to pick up an overlap threshold to be applied by default. However, the performance output (function diagnose_detection) also includes a summary about the overall overlap of the true positives ("overlap.to.true.positives", a number between 0 and 1). If users want to exclude some detected signals with low overlaps they can get the amount of overlap for each detection (in a detection table) with the function label_detection. That would add an extra column that you can use to remove those with little overlap.

Here is an example on how to do that:

#remotes::install_github("maRce10/ohun")
library(ohun)

# load example data
data("lbh_reference")

# create a simulated detection ouput ##
# copy reference
sim_detection <- lbh_reference

# add some time to the first 3 rows so they barely overlap with the original data
sim_detection$start[1:3] <- sim_detection$start[1:3] + 0.1
sim_detection$end[1:3] <- sim_detection$end[1:3] + 0.1

# check peformance (look at the overlap.to.true positives) ##
# for a perfect detection
diagnose_detection(reference = lbh_reference, detection = lbh_reference)

# for the simulated detection
diagnose_detection(reference = lbh_reference, detection = sim_detection)

# remove low overlap ones #
# get amount of overlap
lab_detection <- label_detection(reference = lbh_reference, detection = sim_detection)

print(lab_detection$overlap)

# exclude those below 0.5
filtered_detection <- lab_detection[lab_detection$overlap > 0.5, ]

# check performance again (of course recall will go down!)
diagnose_detection(reference = lbh_reference, detection = filtered_detection)

links: ohun: https://marce10.github.io/ohun/

label_detection: https://marce10.github.io/ohun/reference/label_detection.html

diagnose_detection: https://marce10.github.io/ohun/reference/diagnose_detection.html

Answer 2

I would cross-correlate detected signal with annotated signal (reference) then you can transform you metrics into a question, how good are the two signals correlated, for which the answer could be easier discussed as it follows standard statistics.