Post your "Backup" process

I've seen questions here and there on the forums and thought it would be nice to have a thread dedicated to this. (I searched and couldn't find a similar thread).

Here's my backup process:

1. RAW files are downloaded from CF
2. RAW files are edited in DPP or Lightroom
   
   1. Since DPP saves edits in the RAW file, those edits are backed up when I back up the RAW files
   2. I have Lightroom export the sidecar/XMP file when I export the edits to JPG, that way if I ever start a new catalog and want to re-edit the photos, Lightroom will import the sidecar file that contains the edits
   
   
3. All photos are stored in a hierarchical folder structure by date and file type:
   
   • Year
     
     • Month
       
       • Day (contains JPG files and these sub-folders)
       • RAW
       • PSD (if applicable)
       • DNG (if applicable)
   
   
4. I keep a copy of that data on an external hard drive (I try to keep it updated every month or so)
5. I burn the data to Blu-ray discs and keep 1 copy at my house and another at my office where I'm employed.
   
   • I like this method for long-term storage, meaning anywhere from 6-12 months after the photos were taken, they get burnt to disc as by that time I most likely will not need or want to make any further edits, etc. to the photos.

I have a DIY built NAS that I keep in the basement (FreeNAS) and everyone 6 months or so I make a copy there too, just in-case. I haven't considered an online backup such as Carbonite (despite it being a really great online backup solution) due to the sheer amount of data that I have.

1. RAW and JPG versions of each shot are stored on my HDD at home
2. JPG and selected RAW files are stored on an external HDD at home. I manually back up ~monthly
3. JPG and my RAW files of my favorite shots are stored on an external HDD offsite. I manually back up ~quarterly

Here's my workflow, in three parts - image triage, RAW conversion, image storage.

Image triage

• Swap out CF card in camera*
• Use card reader (ExpressCard or FW800) to transfer images to new folder (e.g. Xmas Portraits RAW) on Desktop
• Import images into a Triage Library in Aperture as referenced images
• Sort through images, deleting the ones I don't want (I wish DxO had a loupe tool and faster loads! Aperture is better for traige...)
• Empty Aperture's trash, checking box to move referenced images to system trash
• Delete the new project folder, UNchecking the box to move move referenced images to system trash

RAW conversion

• Open folder in DxO
• Edit as desired
• Export sidecars
• Process all images to jpgs in new folder on Desktop (e.g. Xmas Portraits)

Image storage

• Copy RAW image folder (w/ sidecars) to hierarchical folder on HDD (organized by year and month, then folder, e.g. Pictures > RAW Images > 2012 > 12 > Xmas Portraits RAW). Note that a copy remains on the Desktop
• Import the folder from the hierarchical location into my RAW Image Library in Aperture as referenced images
• Import the folder of JPG images into the Aperture Library (the default one) as managed images, i.e. directly in the library
• The folder of the JPG images on the Desktop remains until I've copied it to my wife's MacBook Pro (imported into iPhoto for her), then gets deleted from my Desktop
• The folder of RAW images on the Desktop remains until I have backed it up to one external HDD at home, and a second external HDD at work (both with the same hierarchical organization), then gets deleted
• The CF card with the images is now returned to the usable pool, to be formatted next time it goes in the camera (indicated with a slip of paper in the case)

* When shooting with a one-slot camera like my 7D, right after copying the images to the HDD, I'd immediately back the images up to an external drive; with the dual slots of the 1D X, Card 2 serves as backup storage, and I only format Card 2 when it's 80-90% full and all images on it are processed and backed up as above. I currently have a pair of 32 GB 90 MB/s cards that I swap as Card 1, and a third 32 GB 90 MB/s card as Card 2. I also have a pair of 16 GB 60 MB/s as spares, in case I can't get the images processed fast enough.

So, when all is done, my RAW images are in three locations - two external drives in physically different locations, and on my laptop. Then, every 2-3 months, I burn that period worth of RAW images onto DVD(s), and those go in a safe deposit box (in a town that's neither home nor work). The idea is that laptop storage is finite, and although I have not needed to yet, I can delete a prior year's worth of RAW images from the internal HDD as needed to free up space, knowing there are 3 copies elsewhere. The JPG images can easily be recreated, and even then, those are stored on two diferent laptops.

Here is my workflow:

• insert the SD card into the integrated card reader in my laptop
• import RAW files into Aperture on the laptop
• delete delete delete
• make sure that a Time Machine backup has completed to my NAS. (Time Machine is Apple's build-in backup solution: Incrementally backs up the entire HD with a few exceptions)
• on most occasions, trigger a rsync backup of my user folder to the same NAS. This is just in case that the time machine backup get's corrupted. I use rsync with the --link-dest option for versioned incremental backups
• put card back into camera
• format card in camera next time I use it (if I remember)

I still want to add an additional backup to an external Service. My NAS can sync to e.g. Amazon cloud. I just need to figure out the initial load. I know some services allow you to send in a HDD for the initial load, I just need to do some research.

I import my images using a USB 3.0 reader (I recently got a PCI express to USB 3.0 reader link, its amazing how much faster this is now) to a dedicated hard drive labeled images where all my files are stored.

That whole computer gets backed up nightly on my home server.

The image drive mentioned above gets copied to an external USB 3.0 drive ever week when ever I get a reminder to backup my catalog in lightroom. Every month or so that drive gets swapped out with another that I keep off site at a friends house.

So essentially if something happens to my PC I had two local copies available to use. If something happens to all three I have an off site back up that shouldn't be more then a month out of current.

I keep thinking about adding a cloud based back up. But I'm not a fan of subscription services.

Carbonite for all files. Zenfolio for jpegs.

Process

1. Copy RAW images from Firewire800 CF reader to Data Drive on computer. The OS Drive only contains the operating system, so (theoretically) the Data Drive will go to sleep more often and last longer. Also, if the operating system and/or drive crashes (hasn't happened yet), no data is lost. Folders are named [Camera]_[Start Date]_[End Date], eg. Canon-5DII_2013-01-30_2013-01-31.

2. Process images with DPP in that directory.

3. Export full-size JPG to temporary directory on the OS drive.

4. Upload JPGs to SmugMug for display.

5. Delete JPGs. Return CF card to camera (when needed) and format.

Backup

1. When the Data Drive starts to fill (about 6-12 months), copy directories into a "Backup" directory, eg. BACKUP_1, BACKUP_2, etc. Each directory is 250GB (arbitrary size).

2. Copy the Backup directory to an bare HDD via a Firewire800 drive dock. I started with 250GB drives and I'm now up to 1TB drives. The Backup directory is copied to 2 different drives, but each HDD does not have the same content due to the different sizes. For example, HDD1 (250GB) holds BACKUP_1, HDD2 (500GB) holds BACKUP_1 and BACKUP_2, HDD3 (1TB) holds BACKUP_2.

3. Backup drives are stored in an anti-static bag inside a water/fire rated safe. Safe is for other things too.

I still have my images from 2002, which is when I switch from film (Canon Elan IIE) to digital (Nikon CoolPix 5700). I typically upgraded the HDD every few years, along with the computer. Now, I just get bigger drives as prices go down. So, I haven't experienced any HDD failures... yet.

When it comes to backup, some might find that I'm a bit stressed but I have six copy of everything.

1. My PC has 3 drives: 1 SSD for the system and two HDD in RAID1 for the data (that's copies 1 and 2)
2. At home, I have a NAS also in RAID1 (copies 3 and 4). For obvious reasons, the NAS is not in the same room.
3. I have a HDD at work (copy 5) and I use my laptop to sync the home PC with the office HDD.
4. Last year, I bought a 4-year subscription with CrashPlan so now I also have a copy of everything there (copy 6)

RAID HDD is for hardware failure
NAS is for small home disaster
Office HDD is for bigger home disaster
CrashPlan is the cherry on the cake and it syncs faster than the office HDD

I don't know if it's too much or not enough but at least it seems unlikely that I lose all my pictures and the amount of work needed to keep everything in sync is not too much. By the way, I use Total Commander for folder sync. This little tool is cheap and easy to use. Furthermore, you have the real files on your back-up not a big encrypted file that you can't really use without the appropriate tool.

I import files through Lightroom to a small SSD. I use the Lightroom option to make a copy on import on an old drive (WD Green). Once I've given them my first pass of work and uploaded my favorites to Flickr for a backup, I'll move them from the SSD to a more permanent internal HDD. Then, once a month, I run a backup of all my data to an external USB drive that I keep offsite at work.

- Take lots of photos on 7D with CF card.
- Forget about having taken photos.
- Repeat
- When card gets full, curse and swear that card is full, swap to backup card and promise myself to empty it when i get home
- Repeat
- Eventually, remember to take photos off CF card to folder called 'new'
- Batch raw->jpg process in DPP
- Leave all in one big mess of 'new' folder until it's got 1000 photos
- One day every 3 months when there's nothing on tv, try to figure out what I was on when i took the photos and where and what they're of, and sort into some kind of folder system
- Repeat
- Once every few months, read some horror story about how someone lost a hdd and lost all their photos and promise myself to do something about backing up
- Repeat
- Once a year, actually plug in external HDD that's the same size (1tb) as the data drive in the pc, and just do a full mirror

I have a custom built computer so the process is a little bit different.

1. I plug the 5d mk3 into the computer via USB.

2. EOS Utility imports my raw format photos into a folder by date.

3. EOS Utility kindly rotates images.

4. The speed is more than sufficient, a full 32Gb CF never takes more than 10 minutes.

5. The documents, pictures, music & videos folders on my computer are not Windows 7 x64 “libraries”, those have been changed and mapped to actual 4 folders within a 2TB AES 256 bit encrypted volume that resides on 2x 2TB RE4 Enterprise Class Western Digital SATA 3.0 drives in a raid1 configuration. This raid 1 array resides in the computer as letter A drive and is always a red bar in windows, no empty space because the encrypted volume resides there. I must open the encrypted volume with TrueCrypt and the mounted volume is assigned letter D drive. On mounting, the 4 document folders are seen. Drive A and Drive D are not used, I merely mapped the Windows shortcuts to them as document libraries. Without unencrypting the volume with a 30 character password, the documents are secured and nothing could be imported. The raid array can be broken and either of the two drives unencrypted separately if necessary.

6. Adobe Master Suite CS6 is installed on C drive, consisting of two (2) 64Gb Crucial SSD drives in a RAID0 striped configuration for greater speed and power, within Windows 7 64 bit. I use Photoshop 64 bit for editing, occasionally Adobe Fireworks for batch processing (not raw though) and sometimes Lightroom for tethered shooting unless I want live view, I use EOS Utility. These shoots are done directly to the RAID1 array.

7. Photoshop x64 CS6 is using two (2) 500Gb RE4 Enterprise Class Western Digital Drives in RAID0 Striping as cache. The merging of 50 or more raw images in panoramic generates massive cache files of 50Gb, necessitating a large and efficient cache. These items are worked on within the raid1 array and are obviously simultaneously backed up by nature of a raid1 array. Larger projects are moved to the cache array sometimes, then final products moved back to the 2TB raid 1 array. In these instances, the workflow is not saved, only originals and final product.

8. There is occasional encrypted backup to a “bare drive” that is temporarily housed in the Server Chassis which the computer resides in. Offsite backup is not in place, however, I have considered a high tier DropBox account, which would afford many other tools to make the cost worthwhile. This would be facilitated by 50Mbps down/5Mbps up that time warner cable offers, as a flat rate $100 a month “wideband” internet that merges 8 normal cable signals using a new modem made available a couple years ago and in my experience has had absolute, 100% up time over a period of years without ever dropping below 50Mbps down.

9. This system is facilitated by a Foxconn Bloodrage GTI with an Intel x58 chipset, with only 6GB of 1333 ram, a Core i7 Intel and ATI HD 5870 Graphics displayed on three (3) 23” widescreen Viewsonic monitors on dedicated swivel/reach mounts.

10. CD/DVD drives were pulled and replaced with a Corsair Hydro H50 liquid cooled cpu radiator blowing hot air out of a radiator where you would normally see the 3 slot DVD rack.

11. It is raw processing power and I need to upgrade it to SATA 6.0, at least 24GB of ram, better processor, larger drives, etc. but it is performing and backing up well, even as I type these words.


*Sometimes I shoot RAW+Jpeg but it really annoys me to have duplicates when importing. Usually, I work completely in Raw viewing folders and raw images with Adobe Bridge and do not create Jpeg until the end. When I use my Core i3 Asus X54 laptop to shoot tethered or to travel, it is a challenge to open up all network security so I can transfer files from the laptop to the main pc.

I'm not sure but I thought I saw an Apple "computer" back there trying to keep up. lol @ the little beach ball. roflmao.