Method for assessing false negatives for rare sounds in long-term PAM dataset

Question

Hi fellow bioacousticians,

I am working on a needle-in-a-haystack detection problem and am trying to figure out the best way to assess the rate of false negatives. My goal is to detect gunshots in terabytes of PAM data from Central Africa using a template-based detector with monitoR in R, which seems to be working decently well. Calculating the precision (true positives / positives) is a breeze, but recall (true positives / false negatives) is more challenging, since gunshots occur very rarely (there is a gunshot every 1-2 days and the data is continuous). Manually scanning files to see how many gunshots were missed by my detector would be extremely time consuming, not to mention that I would want to see how the recall varies with habitat type, but I can't think of another option to see how many gunshots were missed by my model.

Have any of you ever been in a similar situation with a similar problem? If so, what was the most time-efficient way to get a robust understanding of recall in this context? Or, did you not calculate recall and just spoke about precision?

Thanks!

Answer 1

Unfortunately, if you want to evaluate recall, you absolutely need to know ALL the true events. In theory, you can use any method to find them but the underlying hypothesis is that the method gives 100% correct results. In practice, the "true" events are found by manually inspecting all the files because it is what is considered to be the closest to ground truth (only subject to human error).

You can annotate a subset of your database but it has to be representative of the rest of the database. Then, you can make the assumption that the recall on the annotated database is equal to the recall on the rest of the database.