Is there any software option for diagnosing the performance of an automatic acoustic detection routine? I am thinking of something that can take the output of an automatic detection routine (e.g., a table with the time position of the detected signals) and compare it to a reference table (e.g., a table with the time position of the target signals) and return basic signal detection indices like true positive rate, recall, precision and so on.
The new R package ohun allows to do exactly that (diagnose the performance of detection routines).
It takes a reference table and the detection output and returns a bunch of performance metrics, including number of true positives, false positives, false negatives, precision and recall. It also provides some time related metrics like the amount of overlap to true positives.
I am still working on it so would be happy to hear suggestions for new features.
Edit: to echo @lostanlen, the python package sed_eval for evaluating (diagnosing) sound event detection. It has lots of cool features to evaluate segment-based and event-based detection
I am also looking for tools doing that and seems like people usually do not share them.
Some time ago I tested sed_eval Python tool from Tuomas Virtanen and his Audio Research Group. You can find it here: https://webpages.tuni.fi/arg/index.php/code . I used it on my Raven selection tables but needed to tweak the selection tables before.