Best practice for acoustic data organisation and viewing different datasets in viewer mode

Question

I have SoundTrap acoustic data (ST's were run with a click detector) and I keep two backups of the data in separate hard drives and then work on a copy of these files. I have four devices (two deployed at a time in different locations) and have been generating new binary folders and sqlite databases for each device per deployment.

For the first deployment data I had been working with decompressed SUD files, but am now using the newest PAMGUARD version and processing the SUD files directly.

I was moving between viewing different deployments in viewer mode, giving them a cursory overview, which at first was working as expected. However, when I then tried to reopen data from the first deployment the soundtrap click detector data was coming up as 'no data'. I tried removing the module, re-adding the SoundTrap Click detector and re-importing the SoundTrap Click detector data. It seemed like it was working but then no data was displayed.

When I then tried to view later deployments in Viewer mode again, the data map was showing all the data but no data was loading when I tried to view different sections, no spectograms or click detector data.

I'm wondering if anyone has any advice in regard how to structure the binary folders and databases, would I be better off having one per device and as I get new data adding it to the binary folder/database for example?

Also, does anyone have advice for switching between viewing different databases in viewer mode? It seems as if I have mixed up configurations some how. I was under the impression that everything gets saved to the binary folders and database so I thought that if I had the correct ones loaded (which I made sure I did) that I should be able to view the data as before, however I have clearly made a mistake somewhere.

At this point I think I will start the process again, and reprocess the SUD files, including the first deployment, with the updated version. However, I want to make sure to go forward with the best practice to avoid something like this happening again!

Answer 1

There isn't really a right answer to the question of what is "best practice" since there are compromises at every step. Keeping data on a server obviously has advantages, but can be expensive for big datasets and slow during analysis. I tend to keep recordings (or sud files) on multiple external hard drives, binary, and database working copy on my C drive and where possible a safe copy of those on a server.

The only really fundamental thing is that a PAMGuard dataset (consisting of a database, binary files and probably the original recordings) cannot have data which overlap in time. So if you've two soundtraps out at the same time, their data should NEVER enter the same database and binary folder. If you had a soundtrap out, recovered and replaced it with the same one or a different one, then I guess you could combine the two into one binary folder and view all the data together if that's more convenient for you. If you were putting out the same device for a lot of short deployments, this would probably be very sensible, to reduce the number of separate datasets you have. However for long deployments, there is little advantage.

As an example, I've currently got 15 soundtraps out for a 12 month period, with two service cruises to swap batteries and SD Cards. I plan to keep this as 45 separate PAMGuard datasets, so external drives will have 45 folders of SUD files, and there will be 45 database and 45 folders of binary data. I've considered combining data into 15 datasets, but since individual devices are not going back in the same locations, I find it easier to keep the datasets separate so that they can have the correct calibration and position information with each dataset.

If your data have got a bit muddled (e.g. opening the wrong binary folder with the wrong database), then you should consider deleting files called "serialisedBinaryMap.data" which you'll find with each binary store. These contain a map of the data (basically the datamap) in a form which is more rapidly loaded at startup and saves PAMGuard working through all of the data to make the map. If these somehow get incorrect information, the map may no longer match the data. It's safe to delete these. All that will happen is that next time you run PAMGuard Viewer on that dataset, the map will be rebuilt and the file regenerated, which may require you to go for a coffee, or even a long walk, depending on the size of your data.

Answer 2

I have not experienced your exact problem so cannot speak to what might be going on there but can share how I structure my folders that seems to work for switching often between different deployments and working with multiple instruments both during processing and during viewing.

Within a project I typically have a 'data' folder where the raw data are stored (this may be on an external hard drive or on a server) and then an 'analysis' folder that ideally lives locally on my laptop while I am actively working on that project, but gets regularly backed up to a server (or could be regularly backed up to an external hard drive). If there isn't room on the local hard drive, this could also live on an external drive or server, I just like having it locally for the speed of access.

Then, all my PAMGuard analysis/outputs go in a 'pamguard' folder within the 'analysis' folder. I have separate folders for 'binaries' and 'databases' and I keep my configuration files in the outer 'pamguard' folder. I keep the names of the configuration files for each run, the binaries folders, and the databases consistent across runs so I can keep track of each one.

Here is an example (--> indicates a folder, -is just a file within that folder):

ProjectA  
   -->data
      -->recorder01
      -->recorder02
   -->analysis
      -->pamguard
         -ProjectA_recorder01_noise.psfx
         -ProjectA_recorder02_noise.psfx
         -ProjectA_recorder01_clicks.psfx
         -ProjectA_recorder02_clicks.psfx
         -->binaries
            -->ProjectA_recorder01_noise
               -->20180129
                  -*.pgdf, *.pgdx files...
               -->20180130
               -->20180201
               -PamguardBinarayStoreUIDLog.xml
            -->ProjectA_recorder01_clicks 
         -->databases
            -ProjectA_recorder01_noise.sqlite3
            -ProjectA_recorder01_clicks.sqlite3

This is just how I do it so I'm curious to see if others have different suggested best practices!