The Archive Problem & Frugal NAS

Last week I posted a video about archiving footage to dual layer Blu-rays discs. I thought it was a good idea, but afterward received a flurry of comments over on YouTube about how it might not be the best idea due to the delicate nature of optical media in general. I also received some great feedback via email, including a link to Will Fastie's excellent blog post, "The Archive Problem".

In Will's email he brought up the point that despite the fact that there doesn't seem to be any obvious choice since all electronic media has a definite expiration date, it's good to promote this discussion. If we create any amount of video work and want to keep it, we need to store it somehow. Since there is no one best storage solution, the "answer" seems to be to spread your files over several media forms and store them in different places. When the expiration date of these forms approaches, you must then re-copy them to new media forms. What a pain!

The following is some new information (to me, anyway) that has been brought to my attention since my Blu-ray video. While offering a decent amount of storage to the average user (50GB), the discs are fragile and can lose data in 1-5 years depending on the quality of the disc. I really have no idea how long they will last. For me, they also took an hour to burn. If you have a lot of video to back up (and I do), this process can take too long and tie up computing cycles better used elsewhere. The discs are cheap, but may not be worth the time or peace of mind I lose creating them.

Memory cards, flash drives, and Solid State Drives (SSDs) appear to have the longest life, but cost the most. Cards and flash drives have a built in charge that could last up to ten years if you write to them once and store them away. SSDs are purported to last longer. Right now, a 256GB flash drive costs $70 and an SSD drive of the same size costs about $100. If you need massive amounts of backup space, this is cost prohibitive, especially if you are on a budget.

That returns us to our old friend/enemy, the hard drive. These can you get large amounts of storage space for very little money. A 1TB bare or USB drive will only cost you $60 and that price will keep dropping. Hard drives will fail, however, and it's never a question of if but when. If you use them for archiving, you can't just leave them in a drawer as they must be spun every so often (at least monthly) to maintain data heath.

The best hard drive solution seems to be the NAS or Network Attached Storage. Essentially a little RAID 1 array, these boxes are a mini computer that holds two or more drives that create redundant backup. If you have NAS for 2TB, you have two 2TB drives in the case. If one fails, you replace it and the other rebuilds the archive. Very neat. These are not very expensive and the only real drawback is that both drives are right next to each other. In case of a fire or theft, you're screwed.

I really like the NAS idea. It plugs into your network and your can archive wirelessly and get an instant backup on the second drive. Still, you have to drop at least $300 to get up and running, though getting up and running is stupid easy.

There is a way to craft a "Frugal NAS", which I'd like to discuss. In my little apartment we have a 32" widescreen TV plugged into a cheap desktop computer. We watch all our digital content (Netflix, YouTube, DVDs, Pandora) through this setup, which never gets powered down--just like a server.

As a result, I can do what the NAS box does, since I already have a way to attach hard drives. I can hook up two external USB drives, using one as the archive drive and the other as the archive backup. This allows me to copy files to the archive (wirelessly if I wish) and have an automated backup to the second drive using a free program like Karen's Replicator. If one drive fails, I replace it and KR will restore the data from the good drive.

I really like this idea and feel that it is the most affordable and secure solution. My only question now is when I fill these up do I store them and create a new archive or buy even larger drives and add onto them? And do I want to separate the drives across the network to avoid the fire/theft issue?


XSportSeeker said…
(part 1)
Ehllo Scott, me again! xD

I left out the part about home built cheap NAS systems on my last comment because... well, I didn't want to write a bible or anything.

Like I said before, I'm not in an immediate need for a NAS right now, but I took some time trying to set up one, and I've been researching things a bit on my own for a couple of years.

Earlier this year, I decided to take a jump as I bought a new computer, on trying to use my old one as a homemade NAS build.

I really can't say how things will go in other setups and on different needs, but here's what happened in my case:

- I have zero experience with Linux based systems;
- My PC was relatively powerful... despite being outdated, it was built as a video editing platform several years ago;
- I have ethernet wiring all over my apartment, which is kinda new;

With that said, I tried two different systems: Amahi and FreeNAS. Both are operational systems based on free software standards that allows you to control a homemade "server" of sorts via web controls remotely. They were made to be easily installed, set up, and then managed.

Why did I recommend Synology then, you'll ask?

It's a mix of further easying up management, the size of the package, and me not being able to understand fine control stuff on the software side - spending too much time while not being able to solve problems.

I couldn't summarize everything in a single comment no matter how much I tried, but suffice to say that even if Amahi and FreeNAS (also Nas4Free) were developed to be easily understood and easy to manage, they are still harder to get (specially when problems arise) than a closed commercial system like Synology.

They probably would be enough if I knew the basics of a Linux system and how to manage it, but I don't.

There were several problems that came up because of how Internet connection is set up at home. I have a router/modem that is kinda closed up and cannot be replaced by something that is more open and easy to setup. Depending on how your home network and Internet is set up, it can become a huge time consuming issue if you don't know your way around those.

The more resources you want/need, the more complicated things get. Using them on the NAS basic side (backup and storage alone), it could be easy enough to setup. But if your escalate that to making it a video center, http server, on the go transcoding, expanding access to Internet (outside your local home network), activating services that works and looks like cloud storage, among several other things current NAS devices offers, those can make the install and problem solving process way harder. Waay waay harder I mean.

So at some point you'll have to start considering how much you saved on using a home built NAS, and how much time you have to spent on making it functional.

After about half an year of several problems and frustrations, I ended up scrapping the homemade NAS idea and started budgeting for a commercial solution like the ones offered by Synology.

But that was for my particular needs alone, it's not to be generalized to other cases. Honestly, I've set too high expectations for it. I meant to automate and replace several things I had going on, but it didn't work out.

It's quite possible even that if I land some job that requires a portion of what I needed, a home built NAS would be good enough.

XSportSeeker said…
(part 2)
In any case, good luck! This is a problem that can go all sizes you want, so I'm gonna advise taking care of not going too overboard. For instance, worries about having all your data in a single physical location are worth of attention (with scenarios like house burning down and all), but trying to solve this particular problem can become very money or time consuming... an Internet based solution will require a hefty upstream, paying for cloud services that supports the ammount of data for your needs among others. A non-internet based solution will require shipments of data in whatever medium it is, and someone on the other side to take care of it.

In any case, if you wanna know just how dense and hard things can get on the backup side, here's a couple of hour long video to take a look at:

Don't let the paranoia chew you though. xD

Hope it helps!
Aaron Villa said…
I use 3 redundant medias. A Main drive, a backup of that on a USB drive and another backup on Crashplan. Bought 4 years of Crashplan a few years ago for $140. The price has gone up since then, but the fact that its unlimited to any drives connected to your computer, including network drives. This makes it a great deal. It takes a long time to upload (months for terabytes), but you'll have peace of mind.
Anonymous said…
I've been storing on CDs and DVDs for 15 years and have lost almost nothing. I write two copies to different brands of media and check that the content has been written properly. I think often that is where people go wrong - it wasn't written properly in the first place.

Last year I had a hard drive go down and before I could copy over the back up copy that drive went down too. Hard drives are much less reliable than DVDs in my opinion.

The other mistake people make is in storing too much or huge files that they don't really need.

In the days of standard definition I used to archive a lot of uncompressed edited material and really that was unnecessary as it could have been done as high bitrate MPEG2 instead.
Unknown said…
M-Discs are not significantly more expensive than regular blu-rays, and they test out to last at least a century or two...if not longer. Granted, you only get 25GB per disc, but that still comes out to 1.25 TB of long term archive data (that won't need rotation) for about $220 (or twice that if you want duplicates). Not too bad...something to consider at the very least for backing up finished versions of films/videos long term.