Manual measurements of calls from spectrograms. Should I be concerned with accuracy?

Question

I've been struggling to measure the duration of my field-recorded bat calls well (with decent amounts of reverberation). Manual measurements are common in my study field using spectrograms. Using a graphical user interface, the 'borders' of the bat call are manually outlined - and the duration is calculated. I suspect manual measurements won't be reproducible within and across observers - but this method remains rather dominant in my field. Finding a proper tool and assessing its utility to my bat calls may take a long time however.

I tried looking for a study comparing the effects of using spectrograms on such measurements but failed to find any for bat calls. Should I still go ahead with spectrogram based manual measurements, can anyone point me to related literature?

Answer 1

Taking visual measurements from spectrograms (e.g. by manually placing a cursor on the screen, drawing a selection box in Raven software, etc.) is not an objective method and can result in severe measurement artefacts. On top of this, this practice may also lead to systematic observer biases.

These papers explain the problem in more detail:

Zollinger et al. (2012) On the relationship between, and measurement of, amplitude and frequency in bird song. Animal Behaviour 84: e1-e9.

Brumm et al. (2017) Measurement artefacts lead to false positives in the study of bird song in noise. Methods in Ecology and Evolution 8: 1617–1625.

Answer 2

As highlighted in this paper https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12766, manual analysis where you click on a spectrogram is subjective, depends on your spectrogram settings and cannot be reproduced by others, so I highly recommend that you and anyone else avoid it:) - Use a robust measure like the D-duration instead. The D-duration is the duration of the call between the -10 dB end points of the signal envelope. One way of doing it on your waveform of call x in matlab is:

env=abs(hilbert(x));

%make hilbert envelope
thr=10^(-10/20)*max(env); %define -10 dB threshold
t=find(env>thr); %find samples above threshold
t=t(1:1+find(t(2:end)-t(1:end-1)>1)); % D duration in number of samples 
t/fs  %samples divided by sampling rate to get D duration in seconds

Answer 3

the accuracy of manual measurements may be an issue for you - or not. If trying to compare similar species with short duration calls, then a difference of 0.5ms could be important, but if comparing widely different calls, e.g a Myotis bat (2-4ms) against a Nyctalus (20-30ms), then small differences are unlikely to cause much problem.

One way to address the issue could be by using analytical functions to make the 'decision' for you on where to draw the lines. You can do this using the seewave::timer() function within R. As an example, try this (taken, of course, from Jerome Sueur's book):

library(seewave)  
data(orni)  
timer(orni, threshold=5, msmooth = c(50,0), envt="hil")

Which gives you this output, in graphical form. Or you can save the output as an object to extract the numerical information.

Answer 4

Depends on your accuracy. If you'd like a 'quick and dirty' answer, then go ahead - even though the results may not be all that accurate. You could try to quantify the amount of variation caused within/across observers using synthetic signals (if using Python, scipy.signal.chirp] comes to the rescue. See docs here ).

My own experience is that there's likely a good tool out there which will suit your use-case. The time spent finding the correct tool is typically time well spent in my own experience.

Answer 5

Raven has a set of "robust" signal measurements based on peak frequency contours which are less susceptible to annotator bias; they did a presentation comparing these with traditional measurements that is archived here -

https://ravensoundsoftware.com/knowledge-base/robust-signal-measurements/

Answer 6

One possibility you could use is simply to try to quantify that variation within and across observers. Ensure that each observer is exposed to the same recording multiple times across their review efforts but do not make it clear to them that it is a duplicate, and evaluate the variation on the back end. You can do the same for between-reviewer agreement. It may turn out that for your particular use case and your set of reviewers, you have an acceptable level of variation.