I would like to maximize the file size of my wav files for continuous recordings without significantly sacrificing the performance of PAMGuard analysis.
I realize this depends computer specs (among other things) and that there is no one right answer.
Are there any specific PAMGuard limitations with regards to file sizes, or points where you find processing starts to suffer?
Technically, the way that PAMGuard works, the size of the file should not matter because Java loads the file as a stream (i.e. loaded in chunks) and not all directly into RAM - we regularly analyze files that are > 2GB without issue. However, there are a number of file size related issues that you need to take into consideration;
PAMGuard will attempt to write one binary file per wav file to make post processing data management simpler - if your file is very large the resulting binary files may exceed maximum size and thus multiple files will be written per wav file. Whilst this is not a problem for PAMGuard viewer mode, it can be a pain in analysis with MATLAB/R.
If file sizes are very small (e.g. 10s files written by an AudioMoth) then lots of binary files are written, especially for large datasets. Again, PAMGuard can handle this but it means there is a lot of indexing to do which slows viewer mode (and pretty much everything else) down. There is an option in Sound Acquisition to "Merge Contiguous files" but this is a little obtuse for users.
Large files can be more difficult to handle in programs like MATLAB, however, a little digging into functions such as audioread and there is an option to load a section of file which, just like Java, will use a stream and not load an entire file into RAM.
Personally, I like the default 600MB recorded in PAMGuard as it is a good trade off between having a sensible number of files and each file can be loaded easily into RAM if needed.
I find that 5 min file sizes sampled at 500 kHz (2-3 channels) do well in terms of collecting data in the field. In terms of processing, I find at the moment it has more to do with the computer's amount of RAM, how much you've preallocated for PAMGuard, and where your soundfiles are stored. For best processing, have your files stored on an external hard drive attached to your local machine.
You can change the amount of RAM allocated to PAMGuard by opening up PAMGuard Viewer's .ini file and modifying the first two lines of code. I tend to set up my computer on the conservative side so I set my max RAM to one half of my computer’s total RAM. This allows me the ability to do other tasks with my computer if I need. To set the maximum of RAM to allot to Pamguard, change the value on line 2. To change how much RAM is allocated in general to Pamguard, change the value on line 1.
Example .ini file:
-Xms2048m
-Xmx8192m
-Djava.io.tmpdir="%USERPROFILE%\Pamguard"
-Djava.library.path="C:\pamguard\PamguardBeta_2_02_06\lib64"
-jar "C:\pamguard\PamguardBeta_2_02_06\Pamguard-2.02.06.jar"
-smru
*note lines 4 and 5 are specific to where PAMGuard is installed on your computer so that might be a little different
I'd generally stick to 2 GBytes, sometime less, depending on plans for further processing the wav files.
There is a fundamental limit to wav file file sizes of 4 Gigabytes. This is because the size of the data is stored as an unsigned 32 bit integer in the wav file header and this number simply can't get any bigger than 2^32 bytes. However, some languages (Java being an example) don't support unsigned integers, so are effectively limited to sizes of 2^31 which is about 2 Gigabytes (PAMGuard has a modified wav stream reader since the standard Java one fails with files > 2Gbytes).
If you plan further analysis with Matlab, when you read the files into Matlab, the default audioread(...) function converts the 16 bit integer values in the wav files to 64 bit double precision values, which use four times as much memory. Therefore reading in a whole 2Gbyte wav file into Matlab may require 8Gbytes of RAM (16 for a 4Gbyte wav file). If you've lots of RAM, this may not matter too much and there are ways around this, such as only reading part of the file. However, if you're planning to use software which will read a whole file at a time into memory, I'd recommend a much smaller file size.
At high data rates (>192 kHz), I usually stick with shorter files (1-2 minutes). PamGuard has no problem stitching these together as if one continuous file. You'll find that these files are easier to share with colleagues than the maximum file size. Also, some software will give you problems with very large files.
