We have a large (>500TB) acoustic dataset that so far has remained uncompressed. I am considering going back and FLAC compressing all of our data in order to make data transfers and archive easier and more cost-effective. However, it is active storage and the data are still regularly accessed and used for different projects. MATLAB will read FLAC data using audioread.m and I think PAMGuard will read FLAC files as well. I don't know a ton about data compression, will you lose data processing efficiency if you directly read in FLAC files without uncompressing to WAV? Either because it takes longer to read from FLAC, or because the file is being uncompressed and held in memory? Are there other factors that make FLAC compression not feasible for actively used data storage?
-
3$\begingroup$ PAMGuard will indeed read and write FLAC files. $\endgroup$– user213Jul 8, 2022 at 7:44
-
1$\begingroup$ A couple previous posts on different forms of compression, file types, etc. - bioacoustics.stackexchange.com/questions/344/… AND bioacoustics.stackexchange.com/questions/48/… $\endgroup$– Carly Batist ♦Jul 8, 2022 at 22:36
-
$\begingroup$ @AnnAllen -- If one of the answers below answers your question, please 'accept' the answer by checking the checkmark under the vote tally. See stackoverflow.com/help/someone-answers $\endgroup$– ShannonJul 20, 2022 at 22:18
5 Answers
Any compressed file will take longer to read than a raw uncompressed file simply because the CPU has more stuff to do and, since we are dealing with very large data volumes, this can lead to large increases in processing time. For long term storage, it is also more risky to hold compressed files because a single bit of corruption might make all or part of a file unreadable. If a single bit is switched in a raw file then it alters only one sample (usually).
As a side note, here is what we usually do with high frequency X3 compressed data recorded in the marine environment: The data are roughly x4 compressed - we decompress and then run through PAMGuard (note soon PAMGuard will directly read these files so decompressing will be unnecessary). We run a high false positive rate click detector, noise metrics and LTSA on the full bandwidth data (384 or 576kHz sample rate usually). We also decimate the data to 96kHz sample rate and save that as uncompressed wav files. Once processing has finished we dump the full bandwidth decompressed files and keep only the full bandwidth X3, PAMGUARD binary/database and raw 96kHz wav files.
The reason for doing things this way is that there is very little other than echolocation clicks in the higher frequency band but at 0 - ~50kHz the soundscape is much more complex, with many many different animal vocalisations, environmental and anthropogenic noise (e.g. dolphin whistles, fish grunts, seal scarers etc.). We can be pretty sure that a high false positive click detector, LTSA and noise metrics will extract everything we need in the higher frequency band but automated analysis of the lower band is extremely difficult, so we just save the raw files. That way, we have a much smaller easier to handle dataset that has 99% of what we need but also the archived compressed files just in case. Obviously this strategy is species and environment specific but gives an idea of a data management approach.
-
3$\begingroup$ It depends on the bottleneck. When you have network attached storage, data transfer is slow, and if your processor is powerful, then compressed files load much faster than uncompressed, because less data needs to be transferred. $\endgroup$ Jul 8, 2022 at 9:24
-
$\begingroup$ Yes, good point. $\endgroup$– user213Jul 8, 2022 at 10:11
I have worked with relatively large FLAC data sets (a few TB, not anywhere near 500 TB!) and been very happy with it/have no issues to report. I am not an engineer so am coming at this opinion from the analyst end user side of things. I find FLAC is very easy to work with and don't notice a slow down in processing time, with the caveat that I haven't done any timed tests, and that I am primarily working in MATLAB.
The WISPR system that I've worked with extensively over the last several years records directly to FLAC. In our first years with that project we were converting the FLAC to wav for any bioacoustic analysis. But, in recent years we stopped doing that because we found it unnecessary; most programs can now read FLAC directly (MATLAB, Raven, Audition) and I modified a version of Triton that can make LTSAs directly from FLAC files (I have it forked from the main Triton repository on GitHub and am always looking for people to try it and point out bugs :)).
I have not personally compressed wavs to FLAC using the executable mentioned in this answer, but I have gone from wav to FLAC and back just using audioread and audiowrite in MATLAB. I assume that is not the fastest choice, but I find it is fast enough for a few TB of data. I
Yes, I almost always store large acoustic files in FLAC. It's usually not an issue for smaller files of 10s or even 100s of GB, but files in the TB+ range get unwieldy fast. And moving them is like watching paint dry.
If you have data in the 100s of TB, you will definitely notice your data transfer time significantly decrease...and of course your storage costs. As mentioned above, most acoustic software lets you work directly from FLAC. If it doesn't, it will provide a conversion function. On modern systems, I haven't noticed that much of a performance hit. Conversion is pretty easy to script in just about any modern language.
Besides the decreased storage requirements, FLAC is a lossless compression, it's open source, and it's available for all major platforms directly from the people that maintain it (https://xiph.org/flac/)...or your favorite software. I personally like the idea that it isn't proprietary.
If you are planning to submit acoustic data to NOAA/NCEI, they prefer FLAC.
FLAC is great. I hope to see more projects and application programs using it. Compressing takes a bit of CPU time but uncompressing is easy. Depending on noise level, I get a compression to about 1/2 or 1/3.
Our usual workflow with FLAC is to leave projects that are not being actively analyzed saved in FLAC, which gives us about a 50% file size reduction, while projects that are being actively analyzed stay in WAV format because it's easier to work with, especially in R with seewave and tuneR.
