More workflow thoughts

While I was putting together my post on how I manage my picture workflow and archiving, it got me thinking about an even larger problem, the archiving and workflow of the production server here at Empressr and Fusebox. This post I’m going to talk about archiving workflow.

At Fusebox we have a climate controlled server room with about 3 racks of assorted machines, these are all used for development and staging of applications and Web sites. We have a couple of co-lo racks in a hosting facility here in the greater NYC area and a redundant facility located in a top secret location somewhere in the middle of the US.

First the production environment, we do incremental backups from our rack to our office thru a VPN tunnel every evening, these backups go to a partition on our development server that has restricted access. This partition is synced to another partition on a different RAID and that is backed up to tape. Then, whatever service that falls into needing redundancy (or disaster recovery) is synced to the respected server at our top secret location, yes somewhere in the midwest.

So then on to our development environment, at a high level, we use a combination of Max OS X. Windows and flavors Unix in this environment, so back up gets tricky. All the front end graphics, images, video gets stored on a Mac Xserver that is hooked to a 16TB RAID 5 storage unit. That gets synced on a nightly basis to a Xserver with a 7 TB RAID. The Unix machines also back up to the 16TB RAID. Then this all gets written to tape as a nightly incremental backup, some of the windows machine get backed up directly, some thru a similar process as the Unix boxes.

I find this to be a tedious and expensive process and really want need to change the way we archive. Tape seems too archaic (and unreliable), backing up to the cloud is to expensive, my only thought right now is to have redundant RAIDs making backups of data.

Any thoughts are greatly appreciated, oh and another drawing, this one is a bit sketchy, as I only spent 5mins doing it,

co-lo

My Picture Managment Workflow

A nightmarish problem that just keeps growing and growing is how to deal with all your personal data when it comes to pictures and video. I’ve been using this work-flow for some years now, I thought I should share it and get some feedback. I do plan on making a few changes to it in the near future.

I shoot mostly still images, in RAW format and then process them into JPGs or PSDs. So the first management issue is, once the images are processed, do I delete the originals, no, I  keep it, the same way that you would save the old film negatives for later reproduction, same apply’s here, you may want to develop that RAW image in a different fashion at a later date. I save developed images in the same directory from that shoot. Speaking of file system, I have recently shifted to dumping the days shoot into a single folder with the date, I sometimes will add additional date to the folder as well.

I organize each daily folder into a monthly folder into a yearly folder. I use Expression Media as my cataloging tool and I catalog based on a years worth of images (I use to have one master of all years, until it corrupted one day and I lost all my meta data 🙁 ).

So the hardware flow is as the following, dump to my Pictures folder, (or lately I carry a portable Seagate 500gb drive that I have been using since I keep filling up my Laptop drive, to a 2TB LaCie RAID that I connect to my laptop one or twice a month (more often if I have some really nice shots that I don’t want to risk). I then sync the 2TB desktop RAID with my 2TB NAS RAID that is in a server rack in a climate controlled environment.

Somewhere along in this process I have processed and uploaded some images to flickr, or maybe facebook, or even here to share, I sometimes send full rez, occasionally If I really like the image I will only upload a medium rez and save the high one for just me, I know selfish…) Then every so often (I should do this more often) I will upload my favorite images RAW and JPG to smugmug for archiving. I currently do not allow browsing of this library, but plan on doing so in the future.

So that’s it, yes I know it sounds like a job, and sometimes it feels like it, but It’s heartbreaking to have a hard-drive failure and lose precious memories, especially since I can’t remember shit these days (gotta work on that next)

Oh and it was fun to actually draw a picture, yeah I have skilz