When assessing automated detector performance, how much manual checking of the detections is "enough"?

Question

I have 16 weeks of underwater acoustic recordings (2 recorders, 8 weeks each), and I plan to use an automated detector to detect sperm whale echolocation clicks. I would like to assess the detector performance (precision, recall, false positive rate) but am unsure how much manual checking of the recordings that I need to do to know I have accurately captured the performance.

At the lowest analysis level for this project, I just want to assess presence/absence. But, at a higher level (and thinking back on past work) I'd like to know the answer for this for acoustic density estimation outcomes.

When assessing automated detector performance, how much manual checking of the recordings do I need to do?

Answer 1

Selene, hi. I don't work in marine acoustics, but the amount of manual checking required to assess precision/recall and enumerate your false positives and negatives is probably less than you think. Chambert et al (2018) https://doi.org/10.1111/2041-210X.12910 stated this:

"When false positives occur, estimator accuracy can be improved when a subset of detections produced by the classification algorithm is post-validated by a human observer. We use simulations to investigate the relationship between accuracy and effort spent on post-validation, and found that very accurate occupancy estimates can be obtained with as little as 1% of data being validated".

For my paper on capercaillie https://doi.org/10.1007/s10336-019-01649-8, I checked 4% of the dataset and got useful outputs.

Answer 2

Sampling is a perfectly reasonable strategy. However, you need to make sure that you sample across the different regimes that you might have which could affect performance. As an example, if you are recording in a region that has changes in noise patterns (e.g. pleasure boats on weekends, tidal flow noise, etc.), ideally it would be nice if you had an idea of how your performance varied in these situations.

Answer 3

First, are you planning to train a detector yourself, to use a pre-trained detector or an off-the-shelf model (maybe there is an already existing detector for sperm whale echolocation clicks)? This will influence the amount of labeling you will need to do.

• In the first case, an empirical rule of thumb I found to yield good results is to have 40 hours per sound category (in your case that would be 40 hours of whale echolocation clicks and 40 hours of background soundscape).

• In the second case, using a model that has been pre-trained on Audioset (or even a model that has been trained on marine mammal already) will reduce enormously the amount of manual labeling you will need to do. I got great success training models using only a few hours of sound per category (in my case it was anthropogenic noise, which may be easier to discriminate for a model).

• In the last case, if you already have a model that you are able to use off-the-shelf the amount of labeling you need would just be a few hours (it should be enough to be confident about model performance). In my opinion, the most important factor is to randomly select audio samples (just don't take samples you know contains whale clicks) so you get a balanced dataset (relative to the amount of whale echolocation clicks).