Thoughts on a portable disk "system"?

[peety3]

So....37 days from now, I'll be aboard the Grand Princess heading out on a 7-day Alaska cruise. Last time we cruised Alaska, I didn't take enough "space" - ~80GB of memory cards and ~80GB of internal HD space on my MacBook Pro (with a rented 1Dx in JPEG mode due to an out-of-date Lightroom install). This time around, I don't want a space crisis to be a concern in any way, so I want to assemble a suitable, portable system, and I'd like your thoughts.

1: My MBP is old and tired. The battery should be serviced, I should upgrade the RAM beyond the original 2GB, the touchpad is dying, and I ought to replace the HD with an SSD. Or, I could replace it altogether...though my wife's matching MBP at least has a reliable touchpad. Both have FireWire ports.

2: My Mac mini has a FireWire port. It has Thunderbolt, but I'm already daisy-chaining two 27" Apple monitors on the TB bus, and I question what level of performance I'd get plugging one or more TB drives into the second monitor's TB port. TB drives wouldn't work on my MBP, and they seem to command quite a price premium (OWC/macsales.com puts TB drives ~$75 more than FW800 portable drives, and ~$115 more than USB3.0 portable drives). Therefore, I think it makes sense to stick with FW800 since it's (at least the OWC Mercury On-The-Go units) daisy-chainable, preserving the onboard USB port(s) for memory card readers. I can buy a TB->FW adapter for future computers if desired.

3: I'm thinking about getting one 480GB SSD and two 1.0TB HDDs, always copying the day's memory cards to the SSD, then while at dinner letting an rsync script copy the SSD to one/both of the HDDs. If the SSD might get full, I could delete older files knowing they're on both HDDs (I would audit my rsync script to be sure it doesn't delete files), but if we manage to stay within 480GB I can do my editing at home directly from the SSD. I could put one of the HDDs in our cabin safe or even carry it across the ship to my parents' cabin.

4: I've also considered going with one 240GB (or MAYBE a 120GB) SSD and three HDDs, keeping one HDD as the staging point, and two HDDs as rotating "off-site" backups. The SSD would serve as the rapid-access staging point to get memory cards emptied, then dynamically copied to HDD1. HDD1 would be dynamically copied to HDD2A; when complete, HDD2A would go to my parents' cabin, and HDD2B would come back to be brought up to sync with HDD1. We're probably going to have ~240GB of memory cards with us on this trip.

5: Should I just skip the SSD altogether, and go with the 3 HDD (or just a 2 HDD) scenario?

6: Would any of you bother with a tool such as diglloyd's IntegrityChecker to ensure that the backup copies are 100% intact?

[Kayaker72]

Peety.....you are blowing my mind

As much as I value having photos backed up, I would favor a system that minimizes your efforts while on vacation. I would think having the photos saved on your cards and 1-2 drives would be sufficient. But that is me.

I recently picked up a 2 TB external hard drive USB 3 compatible that was advertised on this site. It is great to have the 2 TB, but transfer speeds are still around 40 MB/sec, I assume because of the write speed of the drive (USB powered). An SSD may be much faster, which would make your life easier while on vacation.

[neuroanatomist]

On a trip I bring a laptop (MacBook Air works well) and a handful of 64 GB USB flash drives, which costs ~$30.

[peety3]

Peety.....you are blowing my mind

As much as I value having photos backed up, I would favor a system that minimizes your efforts while on vacation. I would think having the photos saved on your cards and 1-2 drives would be sufficient. But that is me.

I recently picked up a 2 TB external hard drive USB 3 compatible that was advertised on this site. It is great to have the 2 TB, but transfer speeds are still around 40 MB/sec, I assume because of the write speed of the drive (USB powered). An SSD may be much faster, which would make your life easier while on vacation.

There's some logic I forgot to mention: with a pair of 1Dx bodies spitting out 0.25GB/sec, I'm stocking up on 160MB/sec cards, which aren't cheap. I want to have enough cards to make it through one day's excursions comfortably, dipping into our older/slower inventory (which also happens to be smaller) at the end of the day if needed. There's no way I can justify enough cards to go a week in Alaska without emptying them. I do have a good process/ritual for setting aside the cards that have been read but haven't been backed up, so at least I don't reformat until it's reasonably safe.

So...that means one external device becomes the main copy, and a second external device becomes the backup. I'm looking towards "portable" drives (small form factor, ideally bus-powered) since a 323-square-foot cabin has very little desk/counter space (and I'm hoping to attend the Peter Read Miller Sports Photography Workshop this fall, so I'll have to edit my files each night using this same system). Effort-wise, I know I'm going to have to sit down and feed cards into the reader. After that, I will have a couple scripts pre-written that will handle the file sync to other drives, so I don't see this process being too much work.

[Kayaker72]

Peety....I'd love to hear what you come up with and how it works for you. Given what you are describing, I would be tempted to have a pair of external SSDs/USB3 memory sticks (they are getting close to being the same at this point) as I believe the transfer rate will be faster. That would give you two copies as you are erasing your cards.

[peety3]

I debated and debated, but took the plunge on what I had in mind: a 240GB SSD as a scratch/staging volume, a 1TB HDD as a primary storage device, and two more 1TB HDD as rotating backup devices, all Firewire 800. I also picked up diglloydTools, and I'm incorporating "Integrity Checker" into my workflow via some scripts. Most of the scripted magic is still in (rapid) development, but the general concept is:

Copy the memory card to a designated path/folder on the SSD, and set the memory card aside for safety.
Generate an integrity checkpoint file per folder on the SSD.
Copy the folder from the SSD to HDD1.
Copy the folder from HDD1 to HDD2 (either 2A or 2B, whichever is connected - they'll have a physical sticker to denote 2A/2B, but they're both formatted/named HDD2).
Perform an integrity check per folder.
If the integrity check is successful, move the memory card from "safety hold" to "ready to format".

The fun is now writing the scripts to do the magic in an optimal manner...lots of geekhood going on to analyze the ideal process. More in a week or so.

[peety3]

For the ~2 people still following this thread, I had a productive weekend writing some scripts in perl to automate the magic for the three hard drives.

The package of scripts operates on a 'ticket' concept. In a directory on my mac, I create a simple text file per memory card that identifies what working directory the card is in, the name of the folder where the card's images get dumped, and the amount of disk space consumed by those images. The file name identifies what "stage" the card is in (1=being ingested to SSD, 2=integrity checkpoint, 3=copy from SSD to HDD1, 4=copy from HDD1 to HDD2, 5=integrity verification on HDD2, 6=whole-drive sync from SSD to HDD1, 7=whole-drive sync from HDD1 to HDD2, 8=whole-drive integrity verify on HDD2) or between (A=between 1 and 2, etc.). As the job works its way through my process, the filename is updated to reflect the current stage.

The first script is 'checkpointer'. Once a minute, checkpointer looks for a ticket in stage A, moves the ticket to stage 1, does the integrity checkpoint on the folder, and moves the ticket to stage B. If there happen to be several tickets in A, oh well, only one gets launched per minute (lots of other stuff to happen, so why rush?).

The second script is 'syncer'. Once a minute, syncer looks at all of the tickets waiting. It tallies up how many jobs are active on HDD1, how many jobs are active on HDD2, and gathers a synopsis of what tickets are in what state. It then goes down a pre-determined priority list to find a job that it can start (it will only do one job; one minute later, another syncer will look for another job that can be done):

if any 'B' tickets and HDD1 is not busy, it'll move an B to 3 and work it, bumping it to a C.
if any 'C' tickets and HDD1/HDD2 are not busy, it'll move a C to 4 and work it, bumping it to a D.
if any 'D' tickets and HDD2 is not busy, it'll move a D to 5 and work it, bumping it to an E.
if any 'E' tickets and HDD1 is not busy, it'll move all Es to 6 and re-sync SSD->HDD1, then bump all 6s to Fs.
if any 'F' tickets and HDD1/HDD2 are not busy, it'll move all Fs to 7 and re-sync HDD1->HDD2, then bump all 7s to Gs.
if any 'G' tickets and HDD2 is not busy, it'll move all Gs to 8 and re-verify all of HDD2, then bump all 8s to Hs.

With this priority scheme, if multiple cards happen to get ingested in semi-rapid fashion, syncer prioritizes getting all of the cards to stage 3 before moving any to stage 4 (and so on), as long as the relative drives aren't busy (I don't want to ask one of these drives to read one file while writing another, or I suspect performance will drop like a rock).

The third script is 'dashboard'. Once per second, dashboard re-scans the directory of tickets, tallies up how many tickets are in each state, and gathers a sum of bytes involved in the tickets for each state. As tickets make their way through stages, I can see how many GB are 'in flight', and have a sense of how long a particular task will take. As the tickets arrive at stage 'E' (meaning they've completed their individual checkpoint/sync/sync/verify and now exist in 3 places), the dashboard adds a new line to display which tickets are safely archived.

I recognize that this is all massively overkill, but it suits my objectives: do the dirty work now so I can not worry about the details while on my cruise. I'll essentially just start copying the card into a folder on the SSD manually, then create a new ticket (someday I'll automate that...not this time though) in stage 1. I'll then pull out a sheet of paper and scribble out 101, 102, 103, etc. As cards finish copying, I'll remove them from the card reader and put them next to their respective ticket number, and bump their ticket from 1 to A. We can go to dinner etc., and when we return I'll see exactly which cards are safe to reformat and reuse, KNOWING that I have three copies of the data. As the bulk sync/verify jobs finish (either before bed or the following morning), I'll eject HDD2(A or B) and put it into our cabin safe, then connect HDD2(B or A) and run a dummy ticket F-99 through the process to ensure that the other HDD2 is also brought up to current state. At this point, I'll temporarily have four copies of my data, but the SSD will likely start having files deleted to make space (depends on how aggressively we shoot).

[Scott Stephen]

Peety.....you are blowing my mind

As much as I value having photos backed up, I would favor a system that minimizes your efforts while on vacation. I would think having the photos saved on your cards and 1-2 drives would be sufficient. But that is me.

I recently picked up a 2 TB external hard drive USB 3 compatible that was advertised on this site. It is great to have the 2 TB, but transfer speeds are still around 40 MB/sec, I assume because of the write speed of the drive (USB powered). An SSD may be much faster, which would make your life easier while on vacation.

Yes, to be sure Mrs. Peety is not getting ticked that you are spending too much cruise time with the camera stuff. I know that is a factor in my own vacation photography calculus, anyway!

Hope you have a great time.

[peety3]

Early (pre-cruise) usage testing shows the "overly-complicated" process to be a pretty cool solution. We had a road trip on Friday that yielded ~29GB of originals across four cards. I spent just long enough sitting at my desk to ingest the cards to the SSD and make the respective "tickets" for each card, then while we were out at lunch then shooting some more, the files were copied twice and verified for completeness. When we got home, I ingested two more cards and let them fly through, all on the laptop I plan to take on the cruise, then moved the SSD over to my Mac mini where Lightroom could just tear into those files with reckless abandon.

[peety3]

We're back from the cruise, and IMHO my "little" solution was a fantastic time-saving tool. We shot 349GB, or about 15,000 images, and one day was 157GB alone across 13 memory cards. I spent just enough time to copy the cards onto the SSD and create "tickets" for each card, then went to bed. When I woke up, all 157GB had been duplicated onto two additional drives and the data had been verified. I swapped the last HD for one in the safe, and kicked off the (bulk) second half of the process again, and by the time we got back from our excursion, it had all been copied to a third drive.

As predicted, the 240GB SSD wasn't nearly big enough for the whole week, but it served as a high-speed staging point that could handle two memory cards being written to it, a folder of images being checkpointed for later verification, and a folder of images being copied to HDD1 without seeming to slow down.

If anyone's interested, I'd be willing to share the scripts. They're written in perl, which runs on OS X 10.8 if not older. Drop me a line at templin at templin dot org and I'll send them out. I'm debating whether I want to write my own ingester between now and the Peter Read Miller Sports Photography Workshop in early October (the next major event when I'll be using the drives) - there's a lot of complexity to handle the possibility of two images on the same card but in different folders with the same name, etc.