Data augmentation strategies for bioacoustics machine learning

Question

When training a Neural Network for bioacoustics classification tasks I usually follow a data augmentation pipeline which is composed of:

• Changes in sound to noise ratio: for instance, if my sound track is composed of a bird call and some background noise I would change the SNR so that bird call might appear louder than on the original track.

• Adding some random noise: Adding a bit of white noise to the sound track. My idea in this augmentation is that since each recorder device brands have some specific "base sounds" (I'm not sure of the exact term), adding random noise might reduce the chances of the model overfitting on certain devices.

• Random crop and padding: Since neural networks require a fixed-size input, I crop a part of the audio segment I am interested in and add some padding to the size of the spectrogram is coherent with the model's first layer. Usually, on a segment of 4 seconds I would crop a random segment of 3 seconds and add 1 second of padding.

Some recent papers I have read also use an augmentation technique most use in the visual domain, namely random flip or time inversion define as:

“Inversion of a track along its time axis relates to the random flip of an image, which is an augmentation technique that is widely used in the visual domain.” Guzhov et al., 2021

Even though the authors suggest that this augmentation improved the performance of the model, I am unsure how much sense it make to add this augmentation type. For instance, bird calls always folow the same sequence and using random flip on a bird call might cause the call sequence to be "broken".

Does anyone have any thought on random flip? And more generally, is there other augmentation strategies you are using in your pipeline?

Answer 1

I do not recommend random flip or any kind of time-axis flipping. In the context of animal sound, this could easily change the "meaning" of the signal, in a way which causes problems for your data. Temporal flipping reverses the sequence of calls, but also the temporal pattern within each call. In a deep learning context, the outcome might be "OK" but it might also lead to your target class being less distinctive for the classifier.

Mario Lasseck's 2018 BirdClef paper gives a very clear list of augmentations, and an evaluation of which ones give the most benefit for bird species classification.

In my lab we use data augmentation a lot, most often for bird sounds but for many other terrestrial sounds. We use the audiomentations Python library which makes it easy to apply various modifications. The README gives a good list of the effects and what they do.

For bioacoustic data augmentation I would always recommend:

• Time-shifting
• Mixing in some background "silence" recorded in the wild (i.e. ambient soundscape that doesn't contain your target classes).
• You can even go further and mix together any two sound files, if you're using multi-label classification/detection and you take care that the resulting recording also has the right annotation (e.g. two different species presence). Some people refer to this as "mixup" in image recognition (see also this twitter comment about "maxup"). Combined with temporal shifting this yields a lot of data diversity despite very little signal processing.
• Gentle modifications to the frequency equalisation (this is similar to slightly changing the response characteristics of your mic), achieved e.g. by convolving with an impulse response, or some gentle use of band-stop filtering or SevenBandParametricEQ.
• Closely related to frequency EQ is emulating the effects of distance by adding a bit of low-pass filtering, and/or some echo/reverb using an impulse response. Audiomentations has a specific filter for atmospheric absorption called AirAbsorption.
• Adding a little white noise is OK (and easy) but I prefer to use ambient noise as above. Or consider pink noise, since the white noise (==equal power at all linearly-spaced frequency bands) might dominate in some "quiet" bands of your signal. In speech tech there's the idea of "speech-shaped noise" (having the same spectral profile as the target signal) - perhaps someone should implement this in the software libraries...
• Time masking and frequency masking -- these don't have a very obvious interpretation but are a kind of "dropout" of frequencies or time regions, and seem to help.

In almost all bioacoustic scenarios the above list is unlikely to damage the bioacoustic "semantics" of the signal. Frequency-shifting could change the meaning of a sound; so could rearranging the units within a sequence.

Be careful not to adopt methods directly from image-processing without thinking them through. The scariest example is "image rotation" at randomly-selected angles which would have bizarre consequences for your spectrogram data...

Answer 2

I use OpenSoundscape (Python package developed by Tessa Rhinehart & the Kitzes lab at Univ of Pittsburgh), which has a lot of augmentation/pre-processing tools. Their documentation & tutorials are also amazing for the package & associated functions. If you haven't tried it out yet it's pretty nice.

In particular, I've used the overlay function to "combine" 2 files for, in my specific use case, adding rain to non-rain files. I have recordings from the Malagasy rainforest that I'm using ML models for to detect lemur calls but the model is frequently confused by rain and it's challenging to find enough clips of calls in the rain for the training dataset (both because lemurs just call less frequently when it's raining and rain decreases the detection distance of calls because of the high noise levels it creates). So, I'm trying out overlaying rain onto existing lemur call clips, similar to your "adding noise" bullet point, except that the noise I'm picking is not random, it's targeted. It would be interesting to see how those different strategies worked out though.

In terms of the random flip, it sounds like the spectrogram image would be inverted at random stages within a recording - am I understanding that correctly? I think it would depend on the bandwidth and frequency structure of the sound as well. If you have calls that have high power at low frequencies and low power at high frequencies I'm not sure inverting that would help the model. Perhaps more so with calls that are more variable rather than stereotyped, as you mention.