Is there any software option for diagnosing the performance of an automatic acoustic detection routine? I am thinking of something that can take the output of an automatic detection routine (e.g., a table with the time position of the detected signals) and compare it to a reference table (e.g., a table with the time position of the target signals) and return basic signal detection indices like true positive rate, recall, precision and so on.
$\begingroup$
$\endgroup$
4
asked Jul 5, 2022 at 19:55
-
2$\begingroup$ I don't know of any stand-alone software, but I think there are some MATLAB-based tools that do this for Raven output tables. I am hoping someone more familiar with Raven can chime in, $\endgroup$– seleneCommented Jul 5, 2022 at 20:00
-
2$\begingroup$ Something I struggle with is, "what constitutes an overlap between an automated detection and a reference annotation"? I've written my own code for this, and depending on the motivating question, I've gone back and forth between coding it as "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table", vs. "ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive". If there are software options for diagnosing performance, it will be interesting to dig into how they handle this. $\endgroup$– Cathleen BCommented Jul 5, 2022 at 20:13
-
$\begingroup$ @CathleenBalantic that would be a great additional question. I've asked myself that! $\endgroup$– seleneCommented Jul 8, 2022 at 20:01
-
1$\begingroup$ Thanks @selene, I added this question here: bioacoustics.stackexchange.com/questions/510/… $\endgroup$– Cathleen BCommented Jul 8, 2022 at 21:43
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
The new R package ohun allows to do exactly that (diagnose the performance of detection routines).
https://marce10.github.io/ohun/index.html
It takes a reference table and the detection output and returns a bunch of performance metrics, including number of true positives, false positives, false negatives, precision and recall. It also provides some time related metrics like the amount of overlap to true positives.
I am still working on it so would be happy to hear suggestions for new features.
Edit: to echo @lostanlen, the python package sed_eval for evaluating (diagnosing) sound event detection. It has lots of cool features to evaluate segment-based and event-based detection
answered Jul 11, 2022 at 17:12
-
$\begingroup$ Awesome you are working on it! would be nice to have a summary of species / events etc. per file or in total as a higher level of comparison. Would be nice to have long-term visualization of annotations for overview and quick visula comparison. $\endgroup$ Commented Aug 10, 2022 at 11:43
-
$\begingroup$ Could you elaborate a bit on your suggestions about visualizing annotations? You mean labeled spectrograms? $\endgroup$ Commented Aug 10, 2022 at 17:10
-
$\begingroup$ Visualizing ground truth events next to detections so that you have quick graphical overview where ground truth and detection match or disagree along with the numeric performance metrics. I posted a question related to this problem: bioacoustics.stackexchange.com/questions/1043/… $\endgroup$ Commented Aug 11, 2022 at 9:36
-
$\begingroup$ Hello @MarceloAraya-Salas, i'm afraid there is a problem in your algorithm for counting true positives. github.com/maRce10/ohun/blob/master/R/… As you noticed, you may count true positives multiple times and obtain recall values above 1. You have a heuristic to prune them but it leads to a suboptimal TP. The optimal way to address this is via bipartite graph matching (Hopcroft-Karp) See mir_eval package: craffel.github.io/mir_eval implementation by Brian McFee github.com/craffel/mir_eval/pull/258 $\endgroup$ Commented Aug 16, 2022 at 22:38
-
$\begingroup$ thanks I will look at it $\endgroup$ Commented Aug 17, 2022 at 19:46
$\begingroup$
$\endgroup$
2
I am also looking for tools doing that and seems like people usually do not share them.
Some time ago I tested sed_eval Python tool from Tuomas Virtanen and his Audio Research Group. You can find it here: https://webpages.tuni.fi/arg/index.php/code . I used it on my Raven selection tables but needed to tweak the selection tables before.
-
$\begingroup$ Hi Pavel and welcome, thanks for your response. If your last paragraph refers to the other response, please remove it and add it as a comment of this response when you have enough points to do so (50). $\endgroup$– NoilCommented Aug 9, 2022 at 16:03
-
1