Geeks With Blogs

News

Google My Blog

Catch me at: The List!


My InstallScript Utility Belt My Amazon Wishlist
My Standard Disclaimer


Archives
Chris G. Williams Beware: I mix tech and personal interests here.

Yesterday was pretty frustrating for me. I try not to blog negatively about clients, especially current ones, but I'll make an exception for this. As usual, no names will be used.

Our environment uses separate servers for static image hosting. No big deal. Yesterday, I had to deploy a bunch of static images from development to these servers. Not having done this before, I looked at the server map, confirmed I had the names right (we'll call them Z1 and Z2) and pushed the files out to both servers.

About 10 minutes later, I checked to see if they were showing up properly, and they weren't. Hmm... that's odd, I thought, so I went out to the filesystem to see if I put them in the right place, and they were gone...

Ocasionally my memory sucks, especially when I'm tired. I've been tired a lot lately, not sleeping much, so there you go...  I thought to myself, did I copy these files?  I THOUGHT I did...  oh well, I'll do it now, no big deal. So I copy the files back out to Z1 and Z2.

There's this guy who is notoriously hard to find when you need him. He's not hiding, as far as I know, he's just really busy. It does, however, make it hard to get stuff done in a timely fashion, unless you're living by HIS timeline, which I'm not. We'll call him Joseph.  I need "Joseph" to "flip the switch" on a redirect that takes advantage of the new location of my images. We set up a time, and he's ready to flip the switch. As we talking, I realize I've done something wrong. (Technically, my understanding of what he was going to do was wrong, but anyway...)  I tell him to hang on for a minute, because some of my images arent in exactly the right place. So I run back to my desk and attempt to fix the misplaced images, only to find that they had all been deleted... AGAIN.

Now I'm getting frustrated. So I recopy all the images out there again, making sure they are all in the right place this time. I'm also getting a little paranoid, so I wait about 20 minutes and check again... ok good, still there. Only problem is, now "Joseph" is nowhere to be found. I go by his desk 2 or 3 times, finally resigning myself to send him an email letting him know to fire when ready.

About a half an hour later, I get a call from "Joseph" telling me he flipped the switch, but none of the images are showing up on the site. Thinking I screwed something up in the image path, I start running through possibilities in my head, at which point he tells me that he checked the destination folder and that there weren't any images in it.

So now I've gone from frustrated and slightly paranoid, to just pissed. I know the images are there, I double checked it myself. As I'm telling him this, I'm clicking out to where they should be... and... WTF?!? They aren't there... AGAIN. When I explain to "Joseph" what happened, he starts laughing at me. (This does nothing whatsoever to lighten my mood, though it obviously helped his...)

Then, he tells me to go see Abraham. Apparently "Abraham" has a special job that runs hourly to migrate content from one server to another. Anything that's not on the source gets deleted from the destination. Did I mention, it does this HOURLY? This explains why the images disappeared after only a couple minutes one time, and lasted significantly longer at other times.

So, I go visit "Abraham" and tell him what's happened so far, and what I need to do. He tells me to copy all my content onto Server Y and he'll manually run the job to push it out to Servers Z1 & Z2. Sounds great.

I go back to my desk, copy everything to Server Y and tell "Abraham" to fire it up. 5 minutes later I go back and check, and not only was there nothing (of mine) on Z1 & Z2, but the content I put on Y was gone as well. I tell "Abraham" this and he is now as confused as I am. 

Eventually, my friend "Abraham" gets back to me and says there is a Server X that we must put the content on. His job will then push it to Server Y and then push it to Servers Z1 and Z2. So once again, for the nth time, I push my content out to the server, throw something at "Abraham" and wait.

Success!!

So now I have to find "Joseph" and let him know I am ready for him to flip the switch again (since he unflipped it earlier.)  If you figured out what happens next, you win a cookie (just not from me, I'm fresh out...)

No "Joseph" anywhere. After a few fly-bys, emails and a phone call, I finally track him down. The switch gets flipped and all is right with the world. Almost.

Turns out... the copy worked ok, the switch flipped ok, the images all show up, except for a few dozen that were checked in after this whole process started. Oops.  So now I have to get latest again, push the new content out to Server X and ask "Abraham" to run the job to make sure they show up so we can QA the site.

Only problem is, everything else took so long that "Abraham" has now gone home for the day. (Frustration rising...)  We don't have a backup person for "Abraham" so nobody else can kick off the job. We know it runs hourly, and it's 6 minutes until the hour so I just wait for the hour to come. Once five after the hour has come and gone, I check and the job still hasn't moved the images.

Apparently, hourly doesn't necessarily mean "on the hour." Not wanting to wait for however long it may be before we can test, I opt for plan B. I copy my content out to Server X, Server Y AND Servers Z1 and Z2 and then hold my breath.

Eventually I got tired of holding my breath and went home for the night.

This morning, we had a fresh batch of BugZilla items, but none of them were connected to my image issue. So it worked.

There's more than one moral to this story and I think I've learned my lesson.  What kind of horror stories do you have?  What was your most frustrating day at work?

 

Posted on Wednesday, December 12, 2007 12:02 PM General Interest | Back to top


Comments on this post: bad day at work

# re: bad day at work
Requesting Gravatar...
The day that I was working at a place and having a disagreement with the owner regarding sponsorshp for going to a code camp, and he sent out an email saying "Before we got <client x>, I was making a list of people to lay off", I guess in an effort to give me a message about where I stood on the pecking order maybe?

Anyway, I left and went independant shortly after that...since then they've added a tonne of new employees, but not alot of new clients (all eggs are in <client x>'s basket). I personally didn't care about the veiled threat, but I do care that my friends have their careers put in jeapordy.

D
Left by D'Arcy from Winnipeg on Dec 12, 2007 12:36 PM

# re: bad day at work
Requesting Gravatar...
Hmm - one that ranks up there is where we were migrating about 50GB (SQL data cluster) worth of data over a SAN from our location to one of our customer's remote locations (remote in this case means another time zone). We did a bunch of testing and said we recommend doing it 1) in this order and 2) during these hours based on bandwidth utilization and network prioritization between our location and the remote site. We had worked out in the lab a number of possible ways and come up with a repeatable process that produced successful results each time...we were ready....only the customer says "No, we'll do it another way."

Gracefully (and fully biting my tongue in the process) I tried to explain that "the other way" hasn't been tested recently and when it was last tested it failed miserably and caused the system to go down requiring a major rebuild of the cluster. Customer insists "No thank you, we've heard your recomendation, but we're doing it the other way". Well since "Customer is always right" we bite our lips and flip the switch to do it "their way" and all hell breaks loose about 10 minutes later...the system went down, it did not pass GO, it did not collect $200, it took a dirt nap instead. I turned off my phone as I started the painful process of rebuilding that database cluster and restoring data from tape backup.

Some 6 hours later I remember one of our higher ups coming back to the lab demanding to know why I wasn't answering my phone to tell the customer what had happened....I didn't even justify that with an answer but continued the restore process.

2 days later we completed the data migration - doing it "our way" after rebuilding the system...and the customer rather than thanking us for anything at all demanded instead an "outage report" as to why their critical system was down for a day....blah.
Left by Lou on Dec 12, 2007 12:59 PM

# re: bad day at work
Requesting Gravatar...
LoL, I thought I was reading The Dai.....err....some other site...

I guess there are some benefits to not having to work directly with our customers.
Left by Rick Mogstad on Dec 13, 2007 10:57 PM

# re: bad day at work
Requesting Gravatar...
Yeah we have a Daily WTF at work, but it's usually me sitting in my cube muttering WTF? on a daily basis.
Left by Chris G. Williams on Dec 13, 2007 11:08 PM

Your comment:
 (will show your gravatar)


Copyright © Chris G. Williams | Powered by: GeeksWithBlogs.net