8
$\begingroup$

I am generated LTSAs (long-term spectral averages) using program Triton. I have done this lots in the past, but for my most recent dataset, the LTSA creation is taking WAY longer than ever before... like DAYS longer. I am not working with that long of a dataset or a very different sample rate than used before. But, this is the longest duration of individual .wav files I've every worked with. I'm wondering if the time it takes to read in each .wav is slowing it down? So, my question is, does anyone know why is my LTSA creation so slow?

Current data set - 10 min wav files, nearly except for 15-30 sec gaps between files for file writing, 180 kHz sample rate, 16 bit, LTSA settings 5 sec time average, 100 Hz frequency average. 8 weeks of data, about 7900 individual files. This is taking over 42 hours to run. Output size is about 800 MB.

Previous data sets that were faster - 2 min wav or flac files, continuous, 125 kHz sample rate, 16 bit, same LTSA settings (5 sec, 100 Hz), typically 4-6 weeks of data, 20,000+ files, would run in 8-12 hours.

Data are on an external hard drive directly connected to my laptop. Trying this on a virtual machine directly connected to our server (where a back up of the data are stored) is even slower

I'm using the latest version of Triton on Github at https://github.com/MarineBioAcousticsRC/Triton. I have a 16 GB Windows 10 machine, i7 processor. It is a different computer than I created the previous LTSAs on, but both were 16 GB RAM and i7's.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

The only thing I could imagine is that Laptop memory is exceeding RAM and that there is a lot of swapping going on. The 180 kHz sampling needs about 1/3rd more memory. If this is the case try (as test only) to set the LTSA setting to 1 sec. Unfortunately, I do not use Triton, so I only can give general advice. Closing unused programs the need memories could also help.

Edit: I may need to dig into Triton, to see if it loads the whole file for processing then 10 min 180 kHz is 5/3 more data than 2 min @ 125 kHz. Anyhow, my experience with Matlab is that when it goes really slow, then usually there are memory issues.

Edit2: It seems that Triton loads only data that are needed (here 5 sec) so only difference is sampling 180 vs 125 kHz or 1/3 more.

Edit3: while data are sampled as 16 bit Matlab uses 64 bit as standard. So, a 5 s data set @ 125 kHz will have 1.25 MB on disk and 5 MB in Matlab. @ 180 kHz data will be 7.2 MB. This is not a big issue for a 16GB PC one would think, but >8 GB are already used by system. To really know if PC is running out of memory is to have Memory tab of task manager open and to follow development of memory usage. Especially, if processing is CPU intensive and Matlab defers Garbage Collection (to free used memory), memory can grow as function of time.

Edit4: It may not be the case here, but the presence of NaN of Inf (e.g. divide by 0) will also slow down processing

Improve this answer
$\endgroup$
5
  • $\begingroup$ I don't think its a memory problem because my laptop and the virtual machine are both 16 GB and using Task Manager I can see it's only using 8.7/54% of the memory (with just MATLAB open). If I try that test with a 1 sec average, what would I be looking for? $\endgroup$
    – selene
    Commented Aug 4, 2022 at 18:42
  • $\begingroup$ It does load the whole file before doing the averaging (readaudio(filename), that's what I was worried might be causing the slow down. I was thinking I could probably modify it to speed it up but need to find the time to do that :) And before going through all that effort wanted to see if that would make sense as the problem. $\endgroup$
    – selene
    Commented Aug 4, 2022 at 20:00
  • $\begingroup$ I was incorrect - it is NOT loading in the whole file, only the samples necessary for the 5 seconds to be averaged. I dug in the code to try to change that and realized that is already how it is being done. $\endgroup$
    – selene
    Commented Aug 5, 2022 at 19:30
  • 1
    $\begingroup$ Added more edits to answer to address Matlab $\endgroup$
    – WMXZ
    Commented Aug 6, 2022 at 5:00
  • $\begingroup$ Thank you @WMXZ! I have been running with task manager open watching the memory status and that is why I don't think that is it because it is very steady at just over 8 GB. I'll take a screenshot of it tomorrow and add. $\endgroup$
    – selene
    Commented Aug 9, 2022 at 3:42
2
$\begingroup$

I second @WMXZ. It is likely a memory issue. Do you know what your bit size is? I believe that Triton uses the function audioread, and the standard number format in Matlab is a 64-bit float a.k.a. double (8 bytes), so you need 4 to 8 times as much memory as the file occupies on disk.

I know this isn't really possible in the Triton interface, but if you start digging into the code you can set both your necessary bit size and the data type you want the wav file to be imported as. That could speed things up.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Recordings are 16-bit. So, it could be a memory issue even though it doesn't appear its maxing out the memory, but maybe its maxed out within Matlab or ?? added the bit size and computer specs to my question. $\endgroup$
    – selene
    Commented Aug 5, 2022 at 19:29
  • 1
    $\begingroup$ Yeah, I have this issue with matlab on my laptop all the time, but not on the desktops that have more powerful RAM. For comparison, I run LTSAs in Triton on half hour files, 96 kHz, spanning 2-3 months on a 50% duty cycle. It generally takes 36 hours or more to run on the high ram computers. $\endgroup$
    – etgriffiths
    Commented Aug 6, 2022 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.