12
$\begingroup$

For those who use automated detection and routinely need to evaluate detector performance, what constitutes an overlap between an automated detection and a reference annotation?

Most of my bioacoustics work is with terrestrial species (birds). I've written plenty of code for comparing whether automated detections are lining up with annotations in a "truth" data set. Depending on the motivating question, I've gone back and forth between coding it as, "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table", vs. "ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive". Depending on motivating question and vocalization type of interest, it seems this decision point is another place where error might be introduced. There are likely creative, robust ways to handle this that I'm overlooking. For those routinely tasked with comparing an automated detector's performance to a truth/reference annotation data set, how do you typically handle this?

Inspired by this relevant question: Software options for diagnosing the performance of a detection routine

$\endgroup$

2 Answers 2

8
$\begingroup$

In the R package ohun any overlapping detection is taken as a true positive, as it would be arbitrary to pick up an overlap threshold to be applied by default. However, the performance output (function diagnose_detection) also includes a summary about the overall overlap of the true positives ("overlap.to.true.positives", a number between 0 and 1). If users want to exclude some detected signals with low overlaps they can get the amount of overlap for each detection (in a detection table) with the function label_detection. That would add an extra column that you can use to remove those with little overlap.

Here is an example on how to do that:

#remotes::install_github("maRce10/ohun")
library(ohun)

# load example data
data("lbh_reference")

# create a simulated detection ouput ##
# copy reference
sim_detection <- lbh_reference

# add some time to the first 3 rows so they barely overlap with the original data
sim_detection$start[1:3] <- sim_detection$start[1:3] + 0.1
sim_detection$end[1:3] <- sim_detection$end[1:3] + 0.1

# check peformance (look at the overlap.to.true positives) ##
# for a perfect detection
diagnose_detection(reference = lbh_reference, detection = lbh_reference)

# for the simulated detection
diagnose_detection(reference = lbh_reference, detection = sim_detection)

# remove low overlap ones #
# get amount of overlap
lab_detection <- label_detection(reference = lbh_reference, detection = sim_detection)

print(lab_detection$overlap)

# exclude those below 0.5
filtered_detection <- lab_detection[lab_detection$overlap > 0.5, ]

# check performance again (of course recall will go down!)
diagnose_detection(reference = lbh_reference, detection = filtered_detection)

links: ohun: https://marce10.github.io/ohun/

label_detection: https://marce10.github.io/ohun/reference/label_detection.html

diagnose_detection: https://marce10.github.io/ohun/reference/diagnose_detection.html

$\endgroup$
1
$\begingroup$

I would cross-correlate detected signal with annotated signal (reference) then you can transform you metrics into a question, how good are the two signals correlated, for which the answer could be easier discussed as it follows standard statistics.

$\endgroup$
7
  • $\begingroup$ That would only apply for stereotyped signals. Not all detection methods are trying to find sounds that closely match a template though. $\endgroup$ Jul 10, 2022 at 0:49
  • $\begingroup$ yes, but OP asked for similarity (overlap) with reference signal. $\endgroup$
    – WMXZ
    Jul 10, 2022 at 4:22
  • $\begingroup$ Cathleen mentions that "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table". So it is overlap in time. not similarity. $\endgroup$ Jul 10, 2022 at 23:30
  • $\begingroup$ Marcelo is correct, I meant an overlap in time. An automated detection might overlap one partial note out of a three-note song by a target species. If I'm trying to diagnose performance of the detector, do I count that detection or not? The answer is usually "it depends on the question" so I'm curious about the heuristics others use. (Apologies for any imprecise terminology; I'm coming at this from a biology background.) $\endgroup$
    – Cathleen B
    Jul 11, 2022 at 14:28
  • $\begingroup$ I'm confused but try to understand. Assume you have a three-note song and your detector detected only one note? is that the question? Was the detector trained for this three-note song (is the three-note song in the reference table)? $\endgroup$
    – WMXZ
    Jul 11, 2022 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.