The problems you have probably can't be solved in PAMGuard. The Whistle and Moan detector is designed to locate sections of recordings that likely contain whistle and therefore minimise the data a human observer has to search through. The detector is based on the following (taken from the PAMGuard Help file):
General Principle of Detection Detection is a multi-stage process, the main steps being
- Computation of a spectrogram from raw audio data
- Processing of the spectrogram to remove noise (especially clicks)
- Thresholding to create a binary map of regions above threshold
- Connecting regions of the binary map to create sounds
- Breaking and then rejoining branches of complex regions (for instance, if two whistles cross)
Therefore, harmonics that are high in amplitude (and therefore exceed the detection threshold) will be detected the same as the fundamental frequency. Similarly, if there is a section of the whistle that is low in amplitude, it won't be detected but the section that increases in amplitude will - as there is nothing connecting them (through the detector) they will be counted as two whistles. The detector also sometimes struggles to pick out individual whistles when they overlap as you have said - there isn't anything PAMGuard can do about this (see point 5) and even a human observer would struggle with lots of whistles at once (think about what you're trying to answer - do you need to know the exact number of whistles?). And finally, to solve your low signal-to-noise ratio issue, you'd have to decrease the detection threshold. But, before doing this you should think about why/what you're trying to answer. If you just want to know dolphins are present, picking up the louder whistle should be fine (there'll almost always be ones too quiet to detect, even by a human). If you're wanting to know exactly how many whistles there are, you will need a human observer (but still will probably have some that you're unsure about as they're so quiet). Very quiet whistles could be from very distant animals, so if you're wanting to focus on animals fairly close to your hydrophone, a signal-to-noise threshold might be useful.
So, my best advice it to really think about what you want to know in terms of detecting/counting whistles. There may be a way to solve your problems with Machine Learning, but nothing as readily available and transferable across datasets as PAMGuard. It's a very useful tool and whatever way you analyse data such as this, there will be a need for humans to look at a lot of data to either train an algorithm or verify the findings, and PAMGuard does a very good job at minimising the data a human needs to look at.
Hope this helps!