17

Is there any software option for diagnosing the performance of an automatic acoustic detection routine? I am thinking of something that can take the output of an automatic detection routine (e.g., a table with the time position of the detected signals) and compare it to a reference table (e.g., a table with the time position of the target signals) and return basic signal detection indices like true positive rate, recall, precision and so on.

4
  • 2
    I don't know of any stand-alone software, but I think there are some MATLAB-based tools that do this for Raven output tables. I am hoping someone more familiar with Raven can chime in,
    – selene
    Jul 5 at 20:00
  • 2
    Something I struggle with is, "what constitutes an overlap between an automated detection and a reference annotation"? I've written my own code for this, and depending on the motivating question, I've gone back and forth between coding it as "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table", vs. "ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive". If there are software options for diagnosing performance, it will be interesting to dig into how they handle this.
    – Cathleen B
    Jul 5 at 20:13
  • @CathleenBalantic that would be a great additional question. I've asked myself that!
    – selene
    Jul 8 at 20:01
  • 1
    Thanks @selene, I added this question here: bioacoustics.stackexchange.com/questions/510/…
    – Cathleen B
    Jul 8 at 21:43

2 Answers 2

8

The new R package ohun allows to do exactly that (diagnose the performance of detection routines).

https://marce10.github.io/ohun/index.html

It takes a reference table and the detection output and returns a bunch of performance metrics, including number of true positives, false positives, false negatives, precision and recall. It also provides some time related metrics like the amount of overlap to true positives.

I am still working on it so would be happy to hear suggestions for new features.

Edit: to echo @lostanlen, the python package sed_eval for evaluating (diagnosing) sound event detection. It has lots of cool features to evaluate segment-based and event-based detection

5
  • Awesome you are working on it! would be nice to have a summary of species / events etc. per file or in total as a higher level of comparison. Would be nice to have long-term visualization of annotations for overview and quick visula comparison. Aug 10 at 11:43
  • Could you elaborate a bit on your suggestions about visualizing annotations? You mean labeled spectrograms? Aug 10 at 17:10
  • Visualizing ground truth events next to detections so that you have quick graphical overview where ground truth and detection match or disagree along with the numeric performance metrics. I posted a question related to this problem: bioacoustics.stackexchange.com/questions/1043/… Aug 11 at 9:36
  • Hello @MarceloAraya-Salas, i'm afraid there is a problem in your algorithm for counting true positives. github.com/maRce10/ohun/blob/master/R/… As you noticed, you may count true positives multiple times and obtain recall values above 1. You have a heuristic to prune them but it leads to a suboptimal TP. The optimal way to address this is via bipartite graph matching (Hopcroft-Karp) See mir_eval package: craffel.github.io/mir_eval implementation by Brian McFee github.com/craffel/mir_eval/pull/258
    – lostanlen
    Aug 16 at 22:38
  • thanks I will look at it Aug 17 at 19:46
2

I am also looking for tools doing that and seems like people usually do not share them.

Some time ago I tested sed_eval Python tool from Tuomas Virtanen and his Audio Research Group. You can find it here: https://webpages.tuni.fi/arg/index.php/code . I used it on my Raven selection tables but needed to tweak the selection tables before.

2
  • Hi Pavel and welcome, thanks for your response. If your last paragraph refers to the other response, please remove it and add it as a comment of this response when you have enough points to do so (50).
    – Noil
    Aug 9 at 16:03
  • 1
    @Noil Thanks, I did so. Aug 10 at 11:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.