For those who use automated detection and routinely need to evaluate detector performance, what constitutes an overlap between an automated detection and a reference annotation?
Most of my bioacoustics work is with terrestrial species (birds). I've written plenty of code for comparing whether automated detections are lining up with annotations in a "truth" data set. Depending on the motivating question, I've gone back and forth between coding it as, "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table", vs. "ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive". Depending on motivating question and vocalization type of interest, it seems this decision point is another place where error might be introduced. There are likely creative, robust ways to handle this that I'm overlooking. For those routinely tasked with comparing an automated detector's performance to a truth/reference annotation data set, how do you typically handle this?
Inspired by this relevant question: Software options for diagnosing the performance of a detection routine