5
$\begingroup$

I am interested in class activation maps to interpret the results of classifications by a deep neural network, as shown in pages 173-180 (Figures 5.8 to 5.15) of the thesis of S. Kahl (2020) https://monarch.qucosa.de/api/qucosa%3A36986/attachment/ATT-0/

I have not been able to find the detailed methods they used to produce these maps in BirdNET. I did find some sources and repositories in GitHub to make class activation maps from images, but I would need more information on how to make them out of audio clips and through BirdNET.

Has anyone else attempted this? Thank you!

Improve this question
$\endgroup$

4 Answers 4

Reset to default
4
$\begingroup$

It looks like the OpenSoundscape package provides capacity for generating class activation maps. It's possible you could dig into the OpenSoundscape code and figure out how to apply it to your BirdNET workflow (I haven't tried). CAMs are mentioned in this tutorial and additionally documented here.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you, this is very interesting! $\endgroup$
    – Camille Rondeau
    Commented Jul 21, 2023 at 20:38
  • 1
    $\begingroup$ yes, OpenSoundscape supports class activation mapping of various flavors. I'm happy to help assist you with using them if you get stuck. Unfortunately, though, the full BirdNET model is not publicly available and its not possible to run gradCAM or similar methods using the publicly shared .tflite files. These files are compiled versions of the model, which do not allow full access to the architecture, a necessary component of activation mapping. If you train your own model in OpenSoundscape, you can create activation maps. $\endgroup$
    – Sam Lapp
    Commented Sep 7, 2023 at 3:48
3
$\begingroup$

It is not entirely clear to me how the authors created their class activation maps, but it seems to be using directly some particular layer in their model.

Another approach would be to use LIME. In particular, using the image-based explainers for spectrogram-based audio classification models. This has the advantage of working with any neural network, and the approach is documented - one can use the image guides as a starting point.

Some basic examples of this method being used on spectrograms can be found here:

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Good suggestion. I remember there has been some work on LIME for audio, from 2 different groups: SoundLIME and AudioLIME $\endgroup$
    – Dan Stowell ♦
    Commented Aug 2, 2023 at 8:04
2
$\begingroup$

I played around with a pytorch implementation of this last year but I gave up because my CAMs looked very weird. Not sure how helpful this as as BirdNET is implemented in Tensorflow / Keras? There will be other implementations out there which you might just be able to use out of the box? Potentially though there may be a conflict with what these packages are designed for (images) and the representation used by birdnet (log spectrogram)? I'm not sure how helpful that is!

Improve this answer
$\endgroup$
1
$\begingroup$

It's unfortunately not possible to generate activation maps with the currently available BirdNET models. The publicly available .tflite files for the BirdNET model are "compiled" in a way that only allow inference (creating scores or embeddings on audio). Activation mapping requires full access to the model architecture.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.