9
$\begingroup$

When using automated detection of focal species sounds, when and why should we focus on characterizing probability of detection and probability of false alarm, vs. focusing on precision and recall? What are the pros and cons of either set of evaluation metrics, and are there heuristics for when to use one over the other?

I would like to consider this question under two different motivating examples:

  • Using an automated detection system to detect a species of interest for use in an occupancy modeling framework (i.e., detection/non-detection data are needed within selected survey periods)
  • Using an automated detection system to characterize phenological trends for a species of interest (i.e., number of detections per day over the course of a breeding season)

Inspired by discussion here: Why report SNR, and what SNR is acceptable? (specifically, this answer) and here: When assessing automated detector performance, how much manual checking of the detections is "enough"?

ETA: My experience is in terrestrial bioacoustics (largely an avian lens) which affects my system assumptions. If there is nuance in how to consider this question for terrestrial vs. underwater problems, it will be worth including that in any answers.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ How would you define probability of detection in this case? I struggle with that phrase because it is so integral to density estimation, but I am thinking in this case it is just some normalized number of detections per unit recording effort? Or do you define it a different way? $\endgroup$
    – selene
    Jul 5, 2022 at 20:31
  • 1
    $\begingroup$ Great point of clarification. My terrestrial occupancy lens leads me to define detection probability as the probability of detecting a species at a site, given that the species is present. In an occupancy question, I could estimate detection probability 'p' directly from the occupancy modeling framework, and I suppose not worry about other evaluation metrics (?). Observing some other relevant discussion across the site has made me second-guess myself in considering the varied assumptions we might have about applying that terminology, though. $\endgroup$
    – Cathleen B
    Jul 5, 2022 at 21:07
  • $\begingroup$ So, in your case (sorry...I'm not familiar with occupancy modeling!) 'p' would be based on the detection process as well as some other environmental variables that maybe would impact detectability, because you are getting an overall 'p' from the model? Or would it be solely based on the detection process? $\endgroup$
    – selene
    Jul 5, 2022 at 21:17
  • $\begingroup$ In an occupancy model, detection probability 'p' can be based solely on the detection process (i.e., the parameter is assumed constant), OR can be modeled as a function of other environmental variables (e.g., p ~ time of detection + wind). I suspect there might be some implicit assumptions from terrestrial researchers about this terminology which is why the question and potential terminology differences piqued my interest in the context of bioacoustics. $\endgroup$
    – Cathleen B
    Jul 5, 2022 at 21:45
  • $\begingroup$ Ok thanks - that makes sense, and I am interested to follow this discussion as well :) $\endgroup$
    – selene
    Jul 5, 2022 at 21:49

1 Answer 1

Reset to default
1
$\begingroup$

I guess that depends on the context and the audience of the presentation. I know both metrics can be translated.

I consider detections as hypothesis testing and then a first-kind error is a false alarm (false positive) and a true positive is a correct detection.

I understand that considering everything as classification (detection is binary classification) then different metrics could be used.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.