Tutorials‎ > ‎

Data Storage

posted Dec 30, 2013, 10:42 PM by Sean M. Messenger   [ updated Jul 6, 2014, 2:56 PM ]

Background

Have you had troubles organizing data, pictures, and old files? Issues with failed hard drives or afraid that could happen soon? Losing data from a lack of organization or failed hardware is never good. I have gone through a couple failed hard drives and several backup scares. I have tried out several different backup schemes and have settled on a super redundant setup. I will explain what I am using and why in detail. It is unlikely anyone will have or implement a setup identical to mine, but you can learn a lot from my discussion.

Overview

I have most of my data on my laptop (SONAR) and bulk storage drive (HEPHAESTUS). These are the main points of concern for backup. Arrows in the diagram below show write paths. For example, SONAR backs up to HEPHAESTUS, Crashplan, Google Drive, and Dropbox. 

Diagram of WRITE paths to backup destinations.

We can also consider the read access from each backup location. In the diagram below, arrows indicate access. For example, SONAR can read from all of Google Drive, Dropbox, HEPHAESTUS, ECHO, and Crashplan. At any point in time, a single school document is stored in at least 3 distinct locations (Google Drive, SONAR/NYX, Crashplan). Similarly, any media is stored in 3 distinct locations (HEPHAESTUS, ECHO, Crashplan) where two are local and one has built-in redundancy for disk failure.

Diagram of READ paths from backup destinations.

Crashplan

Crashplan is an offsite backup service. There are four amazing features that make Crashplan the ideal option for anyone. 
  1. Unlimited data backup
  2. Very reasonable prices
  3. Reliable
  4. Backs up direct attached drives for no extra cost
Yay! The last indirectly means I had to pay a bit more to get the Family Plan in order to back up Hephaestus (as a stand-alone OS), but that's worth it. The unlimited backup means I don't have to worry about how many versions to keep or what to backup -- I can just have everything backed up! Just how I need it. I am using Crashplan because you should always have an offsite backup. Unless something catastrophic world-wide or specific to all of Crashplan's datacenters happens, your offsite backup will always be available and secure. Local backups can fail or be lost in a fire/earthquake/flood, but I have not heard of any offsite backups failing as long as you get it with a reputable company (others would be Carbonite and Mozy). They do a good job at ensuring redundancy themselves. This costs $4/month.

SONAR

My main laptop, a Dell XPS, has a 250 GB Solid State Drive (SSD). I replaced the Hard Disk Drive (HDD) that came with it for speed improvements. I had intended to also install the HDD as a second drive in my laptop, but haven't had the need and did not have a solution for overheating (laptop already gets hot). This storage costs $0/month.

NYX

My secondary laptop is a Thinkpad Yoga also with a 250 GB SSD. This is a very lightweight device that I do not store any important documents on. Its primary use is note-taking and quick, portable access to all of my documents. It can see everything on SONAR and the various other backups primarily through use of Google Drive.

Google Drive

I store all of my documents in my Google Drive folder on my computer. My school work, notes, and everything else except program files is stored and synced through Google Drive in a similar fashion to Dropbox. I have 100 GB 1 TB of space that is shared through Google Drive, Gmail, and Picasa/Google+ Photos. This is because I purchased Google cloud space before college when it was $20 for 80 GB per year. What a deal! I got grandfathered in and still have that rate. I recently upgraded to Google's 1 TB plan when they slashed prices, as I was nearing my 100 GB limit. Instead of $20/80 GB/year, I am now paying $120/1000 GB/year. Better rate, more space, and no need to worry about removing data from the cloud. 

So all of my documents (particularly school work) are stored in Google Drive. This gives me access to my documents from any computer that I can sign into my gmail account (personal or school) from. Great for printing out typeset homework assignments when I don't have my computer!

As I mentioned, this also shares space with Picasa/Google+ Photos. I have a selection of photos compiled into albums synced online to Picasa/Google+ Photos. These are some of my favorite or ones that are examples of the stuff I have done. I use this to share photos. Also comes in very handy for linking to pictures for hosting on this website without duplicating them. But photo management is a whole different discussion -- for this discussion, we will consider media as just more data to backup.

Drive is very versatile. I use this for personal syncing and off-laptop access. This costs $9.99/month.

Dropbox

I have 20 GB of space on Dropbox from the Space Race, an educational institution promotional event ran my freshman year in college. This space will go away after I graduate. Dropbox has very reliable data synchronization. That is, any files in the Dropbox folder on my computer are mirrored to an online location. I can share links to these files or folders with friends, or collaborate on folders with others. 

My clinic team, research lab, and several team projects have shared a Dropbox folder so we all have access to the folder and updates are reflected real-time across all our computers. Dropbox does not support simultaneous editing, but will keep conflicting files so you don't lose information. I have found Dropbox works exceptionally well for collaborating on projects. As such, I have a Dropbox folder with all my team projects. 

I also have my phone upload all videos and pictures taken with the camera to Dropbox as a backup. This means I don't have to worry about losing my phone, it breaking, or me otherwise losing my camera photos. This service costs $0/month.

HTC One

My phone that backs up photos and videos to Dropbox and Google Drive. This is a simple, one-way backup that happens as soon as I use my camera. Note that this isn't specific to my phone -- any Android or iOS device can do this. This costs $0/month.

ECHO

This is my Western Digital MyBook 2 TB external hard drive that serves as a redundantly redundant local backup of most of my larger files such as pictures and videos. It has USB 3.0, so access is as fast as to my SSD (as other hardware in my laptop is the limiting factor). Because of this speed and the limited space on my laptop SSD, I also install Steam games to ECHO. This was a one time cost of around $120. Divided over 4 years (how long I've had this specific drive, and a reasonable estimate for the lifetime of any drive), that comes to $2.50/month.

HEPHAESTUS

This is my pride and joy that I could write novels about its utility. I will try to be concise! I ended up starting to run low on space on ECHO due to my RAW pictures and HD video files and wanted to be ahead of the data need. So when ECHO was about 75% full, I started researching for another external hard drive solution. If you need mass storage, you are best off getting a drive to put it on as opposed to hosting it all in the first place off-site. Yes, another Western Digital would work, but I considered it would be worth investing in a more industrial, prosumer, long-term solution that had build-in redundancy.

The next step up from an external hard drive would be:
  •  an additional external hard drive,
  • a larger replacement hard drive, or
  • a hard drive bay.
An additional drive would be bulky and a hassle to deal with. A larger replacement drive is not scalable for when I run out of space the next time. I would go through the same problem in 2 years. 

A hard drive bay, however, had several options for redundancy and was only slightly more expensive than the other options. I decided to investigate the bay approach. A hard drive bay is an enclosure that you can put multiple HDDs in for a modular design. With most of the bays on the market, you can choose how big to make your external drive by mixing and matching HDDs. Some higher end models come with a managing OS that optimizes read/write access and allocates data redundantly (RAID).

Research narrowed in on Synology or Drobo, and there are two general types of bay storage: direct attached storage (DAS) and network attached storage (NAS). DAS drives are connected directly to the computer that uses it, just like ECHO and standard external hard drives. NAS drives are instead connected to the network and give access to the files to any computer (or device) that can access the network. I chose the Drobo NAS because it was not significantly slower access time than the DAS, had a simple interface with a high degree of customization, BeyondRAID redundancy on the disks, and a plethora of apps to increase functionality. The Drobo NAS drive is effectively a stand-alone computer with massive storage and no monitor. You can interface with it from anything on the network and set it up to do backups on its own, stream videos to mobile devices (Plex), and much more. 

In the end, the Drobo NAS device itself cost $500 and I got three Seagate Barracuda 3 TB HDDs at $130 each for a grand total of $890. Using the BeyondRAID redundant backup, I have 6 TB of free space or $0.14/GB. That is cheaper than my Google Drive storage after just one year! Another way of looking at the price is distributing the cost over an expected life of 4 years (very pessimistic!), giving a cost of $18/month. Totally reasonable! Cheaper than my phone's internet access. This would be an excellent stand-alone data storage due to the built-in redundancy and availability of off-site backup apps. So far I am happy with it! I have ECHO backing up to Hephaestus and Hephaestus backing up to Crashplan. This is costing roughly $18/month.

Total Cost

Tallying up all these costs, we find:

Service        Cost / Month
Crashplan      $4
NYX            $0
SONAR          $0
Dropbox        $0
HTC One        $0
Google Drive   $9.99
ECHO           $2.50
Hephaestus     $18
Total          $34.49

At about $1.00/day, this seems cost-effective and exceptionally redundant. It is obvious that my massive Drobo NAS storage is taking more than half the cost, but it is certainly a required expense for me. I need data storage, and that is actually the cheapest way to get it. This will last for years to come. And it even goes to show that cloud storage and offsite backup is actually exceptionally cheap if you research and choose the right choice (*cough* unlimited data *cough*).