It depends you what you wish to collect data on: If you want to know what species of fish and marine mammals there are in different habitats based on sounds in the audio range below 20 kHz most hydrophones will do a good job. However as soon as you wish to speak to noise levels, received levels of calls, probability of detection ect, you will need to know the sensitivity, frequency response and directionality of your hydrophone. You dont need to buy a calibrated hydrophone to know that, but it requires a bit of calibration using a dedicated setup to get that info. If you take care of a hydrophone it will hold the calibration for decades, but you still need to calibrate often, because if you after 3 years find that the hydrophone is partly broken (crystal is cracked typically) how can you know when it happened in the previous three years and hence what data to trust or not..
We have a methods paper where some of the issues around hydrophones are covered:
Madsen, P.T., Wahlberg, M. (2007), “Recording and quantification of ultrasonic echolocation clicks from free-ranging toothed whales”, Deep-Sea Research, Part I, 54, 1421-1444.