33
$\begingroup$

I am curious how people are storing acoustic data in the long term. I know bioacousticians with piles of hard drives filled with decades of acoustic data lying around. But hard drives fail over time. One study suggested that 90% of hard drives survive for three years and 80% for four years. I know of larger institutions that have server farms and climate-controlled rooms filled with drives (swanky!). But this isn't feasible for smaller organizations or individuals. How are you all maintaining your data over time?

Improve this question
$\endgroup$
1
  • $\begingroup$ Is this issue specific to acoustics :-) ? $\endgroup$
    – Noil
    Commented Jul 21, 2022 at 10:24

11 Answers 11

Reset to default
20
$\begingroup$

We have been using a Synology multibay NAS diskstation similar to this one. Additional expansion units such as this can be added to daisy-chain to larger storage. Access to drives can be direct, via network, or you can also have an online access. Requires NAS drives (more expensive than normal internal hard drives). Over the years we have had several drives go bad-- our IT is alerted & they are able to change out the drives without any loss of data.

In comparison, we have lost data in less than a year on other external hard drive storage devices (unlucky!).

Improve this answer
$\endgroup$
1
  • $\begingroup$ I, too, use the Synology NAS. I believe in redundancy, so I also set up cloud storage (through my institution) for my files. But the NAS is where I put all my data first. $\endgroup$
    – Laura Kloepper
    Commented Jul 28, 2022 at 18:24
15
$\begingroup$

I have many hard drives and a backup hard drive for each of those, and for many of them, my collaborators have another copy. Hard drives have failed but the backups have saved the data.

Another system like the Synology system is the Nexsan Beast which can hold up to 1 PB of data and has the option for expansion drives as well. It can also have up to two drives fail without any data loss, and those drives can be replaced as needed. It’s not inexpensive though and my understanding is it has a lifetime of 5-10 years.

Improve this answer
$\endgroup$
11
$\begingroup$

Someone said, the only reliable long-term storages of information are petroglyphs.

I see three aspects:

  • media change over time and modern computers have even no proper interfaces anymore;
  • media size changes inviting recording of lots of noise (i.e. data without any information content);
  • media degrade over time (even petroglyphs).

My suggestions would be:

  • when new more capable media emerge, spend some time in copying all relevant but only relevant old information to new media;
  • remove all data from your storage (declutter your storage) that is older than, say 10 years and nobody has looked into.

AFAIK, continuous copying is the only way to fight physical degradation of media (not only tapes), and one can only copy so much data during a day.

The 2nd suggestion sounds dramatic, but if there is no interest of your data then why keep it?

OK, I know, my noise may be information for someone else, but that sounds very often only like an excuse to not to declutter your data store.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Seconding the suggestion that getting rid of things that are unused is important! Assessing whether the issue is 'no one wants this' or 'no one can find this' is significant, but that is why collective archive projects are getting traction. There is only so much traffic that can result from 'contact the author' lines in publications $\endgroup$
    – Sarah Vela
    Commented Jul 22, 2022 at 14:00
7
$\begingroup$

Thus far, I'm primarily using open public data archives as my way to preserve >5 years, since I know that most "local" solutions will fail or get lost. Openness means that "lots of copies keeps stuff safe", as the saying goes. Good examples would be Zenodo, OSF, Dataverse, Archive.org.

However, an issue with audio data (and other media) is size. Larger audio datasets can be >1TB and public services may be unwilling to provide for such sizes. I don't currently have a general answer to offer for that.

Improve this answer
$\endgroup$
1
  • $\begingroup$ If you are part of NOAA, the NODD is a resource that now supports data archival of large audio collections as active cloud storage on GCP. The caveat is that it must be up to publisher standards, publicly available, and relatively static. While it's still in it's fledgling stages it is available. If interested contact me or your science center IT reps. $\endgroup$
    – Dan Woodrich
    Commented Jul 27, 2022 at 23:34
6
$\begingroup$

Using NAS has also worked for us. It's relatively cheap (a 20 TB hardrive is ~500$), easy to set up, easy to use. You can set your NAS diskstation up in a way that if one or two of the drives fails your data is not lost.

I would still be much happier if we had another longer term archiving option available to us. I have thought about magnetic tape for data archiving. Their lifespan should be ~ 30 years . The cost of the tapes themselves seems quite low, but the cost of the tape drives are a bit too much for us at the moment.

Improve this answer
$\endgroup$
6
$\begingroup$

For long-term storage, how about Blu-ray. Here's a sales pitch from Panasonic:

Panasonic Blu-ray™ discs have an archival life of more than 50 years, in contrast to an HDD's 5 to 10-year and magnetic tape's 15-20-year archival life. This allows data to be stored for longer periods of time without the need for data migration.

More info on Blu-ray(tm) discs at https://panasonic.net/cns/blu-ray_disc/biz_ideal_media.html

Here are detailed info on Blu-ray(tm) and other disc tech by the Canadian governement, including comparisons and instructions for long term care: https://www.canada.ca/en/conservation-institute/services/conservation-preservation-publications/canadian-conservation-institute-notes/longevity-recordable-cds-dvds.html

Improve this answer
$\endgroup$
5
$\begingroup$

Aside from having one master hard drive as the backup of my field recording archive (on a portable hard drive), I've only just begun to archive my 15 years worth of field recordings to DVD. This is a laborous process however, and I am looking to Blu-ray as more efficient method. As other answers have noted, sole reliance on portable hard drives isn't a good idea. When a drive containing a comprehensive archive of recordings failed a few years ago, I narrowly avoided losing the lot.

Improve this answer
$\endgroup$
4
$\begingroup$

Arbimon has pretty much unlimited free storage (though I believe the amount depends on whether it is public or private) and an uploader desktop app you can use to batch-upload in the background which is helpful. Also, Dataverse gives you up to 1TB of storage for free (figshare, Zenodo & OSF are <100GB for free).

I'm also paranoid/old-school and back-up my data on external hard drives as well (see this post: Reliable hardrives for data collection and storage).

Improve this answer
$\endgroup$
4
$\begingroup$

NOAA Alaska Fisheries Science Center acoustics also uses a Network Attached Storage (NAS) system. It is a good compromise between convenience of centralized data (HUGE advantage in sampling versus a bunch of hard drives in a cabinet) and fast enough I/O speed to still be able to read as if it's a hard disk for most purposes (to learn more, look up random reads). If you move to cloud based, you must abandon random reads or program in data staging to your processes which is something rarely thought about for most bioacousticians, has a high learning curve, and is a lot of work.

Improve this answer
$\endgroup$
3
$\begingroup$

Ecosounds is a free archive for storing audio data.

It currently has no limits and has just recently built a remote upload interface.

Improve this answer
$\endgroup$
3
$\begingroup$

Good question, and already quite a few useful answers! I think it is important to differentiate between storage and archiving. Unfortunately web storage media and hard drives seem not to be designed for long-term storage (decades or centuries!). Hard-drive storage facilitates quick retrieval and eventual change of data, but this is not needed for (digital) archives! As said in a former comment, petroglyphs are probably best, but Blu-ray seems to be the only available commercial product right now - we are still waiting for the holographic crystal storing Terabytes for ever and unchangeable in 1cm3! I think memory institutions (museums, and for bioacoustics eventually Natural History Museums) would be the best place to tackle this task, which is far beyond the capacity of individual researchers and/or universities.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ That's a very relevant point re: difference between storage and archiving. I suppose this could also be framed as a question the user of the data could pose to themselves: is this data I will regularly be accessing (storage), or do I intend to simply file this away for an extended period (archiving.) $\endgroup$
    – Tristan Louth-Robins
    Commented Aug 2, 2022 at 2:55
  • 2
    $\begingroup$ Exactly! I have a lot of recordings which I would like to file away, on a pure storage medium but available in a public archive/repository! Notabene: these data would be immune against hacking - data on CD cannot be changed, hence no ransom attacks possible. They also do not need energy during storage. $\endgroup$
    – Klaus Riede
    Commented Aug 2, 2022 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.