Moving Bare Metal at 4am

June 02nd, 2015

Charlie Holbech

Charlie Holbech

SVP, Operations
I never thought I would find myself volunteering to show up at a data center at 4:00am. But after 15 years in web analytics and big data, I finally earned my stripes working the underbelly of the internet.  

As I work towards twenty years of innovating on the internet - two businesses built, thousands of customers served, and a new gig leading engagement for a hosting company - it feels a bit odd to admit that I've never really spent much time in the data center.  Certainly not at 4am, and definitely not by choice.

Data centers have been an indirect but integral part of life, marked by a healthy dose of angst and the occasional shot of amusement (always in retrospect, never in the moment!).  For instance, back in the early days of IndexTools, I remember our CTO Peter getting that bleary-eyed pager call at 3am and rushing out to Budaors to replace some piece of hardware that had gone completely kaput. And that time the power went out during a major storm and the backup generators failed to kick in. Why? Yes, you guessed it: some upstanding pillar of the community had siphoned every last drop of diesel out of the storage tanks!

The fact that the internet relies on actual machines fed with a steady diet of electricity and air conditioning was always very clear to me, but altogether pretty abstract.  As the sales and customer guy, I let the tech guys handle the servers...until last week when I volunteered to get my hands dirty.

Why was I there?

For those Brits out there, The Fast Show's '13th Duke of Wybourne' sketch was the first thing that popped into my head when I got to the main entrance.

I was at our NetAccess location in Parsippany to help coordinate the physical move of one of our racks from one side of the data center to the other - a brand spanking new “data hall” as it were (actually pretty accurate if you see the pics below).  We had a 90 minute window in which to power down 120 servers, unplug network connections, fiber optics and power, move the rack and work the process through in reverse.

I'm told that performing a move like this was a considerable headache in the not so distant past (and in many data centers this is still the case). You'd literally have to disconnect everything, remove the chassis’, switches etc., carry them over to their new home and plug everything back in again. For a full rack that's 6-8 hours work; assuming of course you reconnect everything correctly!

 

 

Fortunately, we were quite literally moving the entire rack, bare metal, switches and all, from a temporary cage to our new permanent one. This method requires a lot less to be disconnected (power, network and exhaust chimney) so the move can be performed far more quickly. It should be noted though, that the estimated overall weight of the full rack though was certainly in excess of 1/2 ton, so to 'wiggle' it into position in its final resting place was definitely a strength test for the NAC team.

 

Once in place we reconnected everything, booted all the servers and confirmed with my team (who I'd been in constant contact with throughout via a Zoom conference) that things we're coming back online on their end. Except they weren't, there was no network.

We were about 75 minutes into the maintenance window at this point and really needed to be wrapping up. One of the NAC guys quietly slipped away and returned with a Visual Fault Locator (a light you shine down one end of the fiber-optic cable helping you determine at the other end which cable is which) and quickly identified that we had an extra fiber-optic cable that had snuck in. We switched cables and were done! Back up and running 2 minutes shy of the end of our maintenance window.

Why did I volunteer?

I've been in operations for years and I'm forever trying to impart the importance of service throughout the teams and companies I've worked with. We, as an organization, must create value and the best way to do this is by providing excellent service. But excellent service is not just a one-way street and it's not solely customer focused. It’s internal too, at all levels of the organization. Sometimes you have to get off your ass and move a rack of servers, right?

I enjoy giving my time, I always have. From helping fix a friend’s computer or building a fence for my Grandmother, to helping a colleague move from one country to another. I always get something out of it and it makes me feel good. You always learn from the experience no matter how small or large the task is. It all comes back to service; providing a good service pleases both the receiver and the doer!

The relocation of the rack had an obvious impact on our customers, and we prepared in all the usual ways to ensure a smooth move. As is almost always the case, it was the engineering team who were scheduled to be up to perform the task. When the engineer who was going to be onsite couldn’t arrange his schedule at the time that was best for the NAC crew and our customers, I jumped at the chance.

It was as much for me as it was for the team. I could actually see what the process was and have a better appreciation and set of expectations for it should it happen again in the future. I thought it would be fun (which it was!). And I saved a colleague from traveling 1 hour 15 min at 2.30am. This, of course, made me feel good and hopefully the team too. ☺

If we want to have a service centered culture it must start within the company. The lengths at which we go to at Packet, to serve our colleagues as well as our clients is superb. It’s part of our DNA already. Can we improve on it? Of course we can, but the foundation is well and truly there and I’m convinced this is a significant part of what will differentiate us from the competition.