What minimum spacing is required to get good bearing estimations from a triangular hydrophone array for low frequency (20-200Hz) baleen whale calls?
Great question, but it partly depends on what you mean by 'good'.
20 - 200Hz calls have wavelengths between 75 and 7.5m. With appropriate algorithms, it is possible to measure time delays to significantly less than a wavelength, so you can often get away with array spacing a lot smaller than you'd expect. In Gillespie and Macaulay 2019 we show that with a good SNR (>17dB) it's possible to measure time delays for porpoise clicks to an accuracy of about a sixtieth of a sample or less than 1/200th of a wavelength. I find I get similar results with Baleen whale calls and even with 20Hz fin whale pulses can get reasonable bearings (10 - 20 degree errors) to these low frequency calls even with an array as small as 1m, though slightly larger would be better.
If you're using PAMGuard for your analysis, there is a simulator which can generate signals in noise from known locations. I suggest you have a play with this and see what accuracy you can get for different SNR's and different array spacing. There are some sample configuration files available to get you started at http://www.pamguard.org/downloads.php?cat_id=3.
As you talk about direction and not localization, I assume that you mean a vertical triangular array.
Well, it depends. It depends on the signal to noise ratio (SNR) and on the type of call.
pulse-type calls (with/without matched filter) that are suited for time-delay-of-arrival direction finding: the larger the better as the angular resolution capability depends on the size of the array
tonal calls that have broad correlation peaks and that are suited for coherent processing and have significant SNR then the maximal distance between hydrophones is half the wavelength for coherent (phase based) processing. The beam-width of such an array will be wide, but the higher the SNR the better the estimation of the direction.