My not so smooth move to Google Apps for Business

Last week we finaly decided to make the move to gmail. Our current email server has been humming along for the last 4 years, it was only a matter of time before something was going to go wrong. We have been using a combination of our own IMAP server running on a G4, yes a G4 xServe and Microsoft’s Frontbridge for spam and virus protection. Initially the transition was smooth, a few DNS changes and we were up and running on gmail, sweet. Then came the migration of my 15 years of email, not a problem Google has a tool for that, it did take a while (hours), I had 10gb of mail (I delete a lot, sorta my workflow). Everything looked OK when I logged in from a web browser. Then the fun started, I opened my Mac Mail program, created a new account for my gmail and let the IMPA synch begin.

I left for some dinner, when I arrived back, I had a message that I ran out of disk space, I panicked a bit since some years ago I had a similar experience and my MacBook was never the same. I quickly dumped a bagillion pictures to my backup (I have been meaning to do that anyway) and let the sync finish. But I was curious, I knew I had about 10gb of mail, and when I started the sync, I checked and I had about 25gb available on my drive, hmmm something is not right so I started digging around to locate the culprit.

First I did a i on my old IMAP and the new gmail IMAP directories, wow gmail was 3x larger! (gmail on the left)

Pretty strange! so more exploration, looking around my Mac Mail app I noticed this,

Ah-ah, that makes sense, well sorta, why would Google have 3 copies of the same email, it’s tripling my mail? and upon a little further looking, I found that emails were sometime 4x and 5x repeated (mostly due to some messy old filtering on my part). wow. Not very Googley. I did a little thinking and in a bold move I selected the migrated “Lable” in my browser based email and deleted the reference to that label, re-synced from the Mac client and low and behold my gmail box fell to 20gb.

Why did this happen, well I have no idea why there is a migrated folder for the email, that’s a Google thing (and customer service did inform us to hide the label, 3 days after we filed a ticket). As the years evolved using email, thru many different clients, I created a lot of sub directories on my old IMAP server, this at the time, was a good way of dealing with hundreds of email a day, I had filters that would move junk, 2nd, 3rd and 4th level email messages so the the primary important stuff was findable.

Google while they support IMAP, does not work the old way, the approach to IMAP is by creating lables, and you can use filters to apply lables, but the messages never get moved into subdirectories. The apple mail client is configured to work like the older style IMAP servers and would make a copy of the emails in your gmail Inbox into the subdirectories, leaving a copy in your Inbox and then duplicating the “migrated” and “allmail” labeled emails into those subs, as well as making additional copies into your other IMAP subdirectories.

So when I have some time I am going to remove most of the older IMAP folders and replace them with Mac Mail smart filters, I have used smart filter pretty much exclusivly for the last few years, but still left the old folder structure inplace, just because. Then I plan to create similar filters to match the desktop client.

Yeah some may say why use the desktop client, for me, it is still a lot faster to scan and delete (a workflow I have been using for a long time) in the client than in the browser. maybe over time I will change my workflow to browser only.

Mystery solved.

My Advice, once the migration of your email is complete, clean up any of the IMAP directories using gmail in your browser, remove the migrated label (I haven’t removed the “allmail” label yet) and then sync, it will save you a massive headache and time.

Update, I also made these recommended settings from Google

More workflow thoughts

While I was putting together my post on how I manage my picture workflow and archiving, it got me thinking about an even larger problem, the archiving and workflow of the production server here at Empressr and Fusebox. This post I’m going to talk about archiving workflow.

At Fusebox we have a climate controlled server room with about 3 racks of assorted machines, these are all used for development and staging of applications and Web sites. We have a couple of co-lo racks in a hosting facility here in the greater NYC area and a redundant facility located in a top secret location somewhere in the middle of the US.

First the production environment, we do incremental backups from our rack to our office thru a VPN tunnel every evening, these backups go to a partition on our development server that has restricted access. This partition is synced to another partition on a different RAID and that is backed up to tape. Then, whatever service that falls into needing redundancy (or disaster recovery) is synced to the respected server at our top secret location, yes somewhere in the midwest.

So then on to our development environment, at a high level, we use a combination of Max OS X. Windows and flavors Unix in this environment, so back up gets tricky. All the front end graphics, images, video gets stored on a Mac Xserver that is hooked to a 16TB RAID 5 storage unit. That gets synced on a nightly basis to a Xserver with a 7 TB RAID. The Unix machines also back up to the 16TB RAID. Then this all gets written to tape as a nightly incremental backup, some of the windows machine get backed up directly, some thru a similar process as the Unix boxes.

I find this to be a tedious and expensive process and really want need to change the way we archive. Tape seems too archaic (and unreliable), backing up to the cloud is to expensive, my only thought right now is to have redundant RAIDs making backups of data.

Any thoughts are greatly appreciated, oh and another drawing, this one is a bit sketchy, as I only spent 5mins doing it,


My Picture Managment Workflow

A nightmarish problem that just keeps growing and growing is how to deal with all your personal data when it comes to pictures and video. I’ve been using this work-flow for some years now, I thought I should share it and get some feedback. I do plan on making a few changes to it in the near future.

I shoot mostly still images, in RAW format and then process them into JPGs or PSDs. So the first management issue is, once the images are processed, do I delete the originals, no, I  keep it, the same way that you would save the old film negatives for later reproduction, same apply’s here, you may want to develop that RAW image in a different fashion at a later date. I save developed images in the same directory from that shoot. Speaking of file system, I have recently shifted to dumping the days shoot into a single folder with the date, I sometimes will add additional date to the folder as well.

I organize each daily folder into a monthly folder into a yearly folder. I use Expression Media as my cataloging tool and I catalog based on a years worth of images (I use to have one master of all years, until it corrupted one day and I lost all my meta data 🙁 ).

So the hardware flow is as the following, dump to my Pictures folder, (or lately I carry a portable Seagate 500gb drive that I have been using since I keep filling up my Laptop drive, to a 2TB LaCie RAID that I connect to my laptop one or twice a month (more often if I have some really nice shots that I don’t want to risk). I then sync the 2TB desktop RAID with my 2TB NAS RAID that is in a server rack in a climate controlled environment.

Somewhere along in this process I have processed and uploaded some images to flickr, or maybe facebook, or even here to share, I sometimes send full rez, occasionally If I really like the image I will only upload a medium rez and save the high one for just me, I know selfish…) Then every so often (I should do this more often) I will upload my favorite images RAW and JPG to smugmug for archiving. I currently do not allow browsing of this library, but plan on doing so in the future.

So that’s it, yes I know it sounds like a job, and sometimes it feels like it, but It’s heartbreaking to have a hard-drive failure and lose precious memories, especially since I can’t remember shit these days (gotta work on that next)

Oh and it was fun to actually draw a picture, yeah I have skilz