13
$\begingroup$

In computer vision it seems that image normalization is very important to keep the gradients "in check" (detailed description here). However, I was wondering if it was also the case in audio deep learning? Is normalization a necessary step for training efficient models?

The package Audiomentation offer the possibility to do peak normalization with the normalize function which:

Apply a constant amount of gain, so that highest signal level present in the sound becomes 0 dBFS, i.e. the loudest level allowed if all samples must be between -1 and 1.

However, I believe that peak normalization can induce some problems. For instance, if some audio files have some impulse noise present (close to 0 dBFS), the signal would end up with a very much lower-amplitude signal than expected (i.e. compared to an audio file not containing any impulse noise).

$\endgroup$

2 Answers 2

10
$\begingroup$

Yes, you should definitely normalise audio data before supplying it to a deep learning model. But don't use "peak normalisation" and don't normalize each sample individually: use either (a) mean-and-variance normalisation, or (b) PCEN.

More detail follows...

Firstly, your question refers to "ML model" but it's only deep learning, not machine learning in general, where this matters, because of the gradient descent used in training. Many other algorithms (I'm especially thinking of random forests, and similar) are not affected by normalisation.

The most common approach in audio DL is I think the same as the standard approach in image and other deep learning, which is to use the mean and standard deviation (measured on the training set) to normalise the data. (There's a mathematical justification for this by LeCun et al (1998), ``Efficient backprop''. The main lesson is: it's easy for us to calculate-and-remove the statistical moments in the data, so we should do that and let the training algorithm concentrate on the harder part -- it will thus train faster.)

Extra detail: Note that this "mean and variance" normalisation is typically not a per-sample normalization. Instead, it involves adjusting the values of every sample by fixed scaling and shifting constants so that the mean of the entire dataset is 0 and the standard deviation of the entire dataset is 1. (For instance, if the mean pre-normalisation value across all samples is 0.15, you would subtract 0.15 from every sample regardless of the sample's mean.) This avoids the issue of quiet or empty samples being normalized to have maximal values and becoming loud and noisy

You should not use peak normalisation. (I don't know why Audiomentations implemented it! The documentation refers to exporting audio, not to training.) Clipping is not a concern for us when doing DL, because the processing in deep learning is usually floating-point, and going beyond [-1, 1] is no problem. On the other hand, peak normalisation is likely to make gradient-based learning worse, because it does not help us to control the mean or the standard deviation of the input data--in fact it's likely to do the opposite since the max and min value can vary so aritrarily. For example, the "impulse noise" issue mentioned in your question.

Aside from the standard method, there's also "PCEN" (per-channel energy normalisation), which is designed specifically for audio and takes a different approach. PCEN performs a kind of dynamic gain control (a bit like a "multi-band compressor" audio effect) which is designed to ensure that the energy values within each frequency band are mapped to a range that is "well-behaved" for ML. Lostanlen et al (2018) give a detailed analysis of this issue, arguing for PCEN. I've seen quite a few bioacoustic DL projects using PCEN as the preprocessing (and no other normalisation) and it seems to be a relatively safe low-regret option.

Extra detail: Modern deep learning networks often have normalisation layers embedded inside them (with names such as "BatchNorm", "LayerNorm"). If these are applied at the input layer then they might sometimes be equivalent to pre-normalising your data... but not always, so I do not recommend skipping the normalisation.

$\endgroup$
3
  • $\begingroup$ I came across some papers that mean - std normalize the dataset on the frequency bins of the spectrogram. Is this something you consider? Or do you simply normalize the array itself? $\endgroup$ Jul 27, 2022 at 6:50
  • 2
    $\begingroup$ Yes, doing it per-frequency-bin is pretty common, and can be a good idea. Whether to do it per-bin or on the whole matrix seems to depend somewhat on the situation. (PCEN is effectively doing it per-bin) $\endgroup$
    – Dan Stowell
    Jul 27, 2022 at 7:12
  • 1
    $\begingroup$ I've been summoned! :) I'm upvoting this answer and will add that i have written a paper for EuroNoise 2021 explaining why i think PCEN has potential in bioacoustics, what are some known applications, and what are the next steps. hal.archives-ouvertes.fr/hal-03381500/document I have proposed to describe the PCEN mechanism as "self-calibration" of the sensor so as to reduce the risk of ambiguity with forms of normalization whose parameters are shared across test set samples (e.g. batch norm). $\endgroup$
    – lostanlen
    Sep 10, 2022 at 8:53
4
$\begingroup$

Dan Stowell’s answer is excellent. I will just clarify that mean-and-variance normalization he described is typically not a per-sample normalization. Instead it involves adjusting the values of each sample with fixed scaling and shifting so that the mean of the entire dataset is 0 and standard deviation is 1. This avoids the issue of quiet or empty samples being normalized to have maximal values and becoming loud and noisy.

$\endgroup$
4
  • 1
    $\begingroup$ Thank you Sam. If you want to, feel free to "edit" my answer $\endgroup$
    – Dan Stowell
    Jul 27, 2022 at 7:12
  • 2
    $\begingroup$ Welcome Sam! Thanks for your comment! On StackExchange, this kind of post should be a comment addressed below the answer you are talking about. If you cannot comment yet, see here. $\endgroup$
    – Noil
    Jul 27, 2022 at 8:25
  • 1
    $\begingroup$ Thanks, should I delete this answer? $\endgroup$
    – Sam Lapp
    Jul 29, 2022 at 23:29
  • $\begingroup$ @SamLapp I would say so (or a moderator), but several people upvoted your questions so you may want to wait for a bit in order not to lose some 'priviledges' (comments, etc) ;-) . A tip: tag the name of the person you are talking to in the comments (@Name) so that they receive a notification of your reply (I saw your reply by chance actually :-) $\endgroup$
    – Noil
    Jul 31, 2022 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.