In early 2016 Packet had just one location: a lovely top tier facility about 50 miles from our Manhattan office in the rolling hills of Parsippany, New Jersey. If something really needed to get done, we could always grab a Zipcar, brave the Holland Tunnel and roll on out.
Next week we’ll open our fourth data center in Tokyo (JP), after adding facilities in Sunnyvale (CA) and Amsterdam (NL) in March. With that growth has come a ton of learning, and my goal with this post is to share some of the tips that looking backward have helped us a lot.
The Abstraction of Infrastructure into an Experience
Over the last few years, the concept of "the cloud" has made its way deep into our psyche. Without a doubt the idea of cloud computing has succeed in obscuring the physical aspect of infrastructure. Everyone knows the joke: "The Cloud is Just Someone Else’s Computer."
But in reality, I think the cloud is more of an experience - a promise that someone else will deal with the mundane work (we would call it an art form) of procuring gear, racking and stacking it, connecting it to physical networks, and repairing / replacing these things as needed.
The accepted consumer experience of the cloud is now much more like buying a cup of specialty coffee: no need to grow the beans, roast them with expertise, grind them to proper grit for your expensive La Marzocco espresso machine. Just hit an API or login through a nice portal and deploy global infrastructure while sitting in your pajamas sipping that cup of coffee you just brewed from your auto-delivered Keurig pod.
The Craft of Physical Infrastructure
But of course there is a huge amount of nuance that goes into global cloud infrastructure, just like there is for amazing coffee or anything else. While we celebrate the engineers who design and build our IPv6 capable network, and pay homage to the specifications of the latest high-density server line, it’s rare to get excited about the logistics of moving things around, paying for them, and making sure you don’t get screwed in the process.
Unless you’re me :) As the CFO, the job falls on me to figure out creative solutions to questions like:
- How can we install those new MicroCloud chassis in Amsterdam?
- What’s the best way to insure and ship $1 million in gear to Japan?
- How do you set up an international subsidiary entity? Should you?
- How do you get the best cost of financing for millions of dollars of capital equipment?
Happily, as a former financial trader and full-time process junkie, I love the challenges involved in finding solutions to these questions while adding substantial value to our bottom line (shhh - don’t tell the engineers that it’s not all about code!). Here’s some of what I’ve learned:
As we prepare to spend tens of millions of dollars in the coming months to buy equipment for client demand, nothing (other than revenue) is more important to our bottom line than finding the best financing. We could, of course, use some of the cash we just raised in our Series A funding round to buy what we need, but we would be out of cash quicker than Supermicro can deliver you their latest catalog!
The only way to do this is to lever up a little bit and stretch out the payments over time. Enter the lovely world of equipment financing.
Early on, as a startup, the decision was easy: with no real customer traction, steady expenses, and only visions of big revenue we were limited to a couple firms that had experience in the cloud business and were willing to take our credit risk. The best we could get with our balance sheet were relatively expensive FMV (fair market value) leases.
In our FMV structure we were basically paying the entire purchase price (plus interest) over a 3 year period and then would have the right to buy the leased equipment at FMV, likely an additional 30-35% of the original purchase price. So for the lessor, basically built in profit on a risky leasing deal secured by a bunch of servers: pretty expensive but an absolutely worthwhile way to grow from nothing more than an idea. We are grateful to the firms that took the risk on us way back in 2015!
However, on the back of our Series A and with some brand name investors, we have seen our options improve substantially. We’re currently in discussions with over a dozen financing partners (and I’m quite sure in posting this blog I’m going to hear from a few others!), and some are open to capital leases. In a capital lease Packet would own the equipment outright at the end of the lease, allowing all those servers, routers and IP addresses to show up as assets on our balance sheet and improve our financial standing over time.
The major obstacle we still need to figure out is our international leasing, which represents a growing part of our business. Domestic (US) lessors are somewhat hesitant to offer lines for equipment going abroad due to the obvious difficulties of getting that equipment back in a default scenario.
I am confident we will have much more clarity in the coming weeks and I look forward to reporting back in a followup post.
Installing physical hardware and network equipment globally requires the use of trusted local “Remote hands” services, as it is usually impractical for a small company to have dedicated employees with the right skills available globally. As such, companies like Packet, and other global infrastructure users, rely on a loose network of sought after onsite engineers to extend our footprint in far away lands.
While the hourly rate for these experts can feel expensive, we get the benefit of local knowledge (e.g. Where can you get Sfp optics at 2pm on a Saturday in Amsterdam??) and only pay for the time and tasks we have in each market.
The remote hands business is based on a huge amount of trust. While you can confidently trust the employees of a major colocation provider, it is well worth getting to know your providers in person. We’ve formed great relationships in each region, such as with remote hands startup Noc Noc.
Another interesting option for us has been to centralize as much of our “remote” work as possible. As such, we use integrators to build complete compute racks to our specifications and then ship fully loaded racks to the remote data center. This reduces another variable in the equation, which is critical as you scale from an order a month to a few orders per week. No amount of my favorite vice (let’s just say it’s ice cream) can lessen the pain of keeping air traffic control on 20 shipments of gear going in different directions.
Shipping and Insurance
Speaking of shipping! When you start thinking about sending lots of heavy equipment all around the world, you realize how expensive that can be in terms of both shipping charges and insurance premiums.
Early on we let the manufacturers just ship according to their preferred shippers. However, when we took the time to check things on our own, we found that we could get much better rates by selecting our own partners - upwards of 50% cheaper than the standard shipping options. As a result of this, we have moved to one preferred shipping company.
Our first ever shipment was about 20 boxes of Juniper gear going from a reseller in Hoboken to Amsterdam. I was texting back and forth with our rep and it became clear we needed to palletize and shrink wrap all of these boxes and the only problem was - we had no pallet or shrink wrap to speak of. After running concentric circles around the reseller location using google maps, I finally came across a moving company who was willing to sell me one pallet and a roll of shrink wrap. They even picked me up from the Path station and dropped me off to wrap up all the boxes - and I am still around to tell the story. Happily we’ve improved the process!
As far as insurance, this is another hidden cost driver. To help us manage this and reduce expenses, we negotiated a blanket policy with a major provider to cover all of our global shipping requirements. We just keep track of everything we ship and then we pay a premium after each quarter end. Nice, simple and effective - words I can’t necessarily apply to our health insurance (but that is for another post).
Due to our business needs, we expanded internationally very quickly. We therefore had to quickly research all the implications of setting up overseas operations, including the logistics of sending equipment through customs as well as things like taxation. After selecting Amsterdam to be our first global location, it became clear that we were going to need to set up a subsidiary in the Netherlands in order to reclaim VAT (value added tax). Every shipment of equipment into the Netherlands was going to come with a lofty 21% VAT requirement that we would have to pay before the goods could clear customs and make their way to the data center.
With the help of our shipping partner and a local accounting firm, we were able to navigate the process and get things properly set up. Now it’s pretty seamless: we pay VAT as we go and get our refund roughly every quarter. As the velocity of shipments to Amsterdam increases, we will eventually apply for an Article 23 permit which allows us to import without paying VAT.
Upon our decision to open our fourth location in Tokyo, we immediately set out to create our K.K. company (local corporate entity) there. This is necessary as we build out our presence and start to generate revenue and pay expenses locally. It was also required for the same issues we faced in The Netherlands in terms of importing equipment. We expect to be able to eventually take a deduction against the import duties charged in the customs process.
I hope this gives you the sense of some of the behind the scenes topics that we have had to address over the past year or so. There are always areas within an organization where you can tighten the belt and reduce costs if you just take the time to focus and then question everything. If you are interested in learning the gory details of any of the issues I covered, please reach out to me directly and I'd be happy to give more context on our decisions and experience so far.
Look for a future post where I will talk about my experience of leaving a steady banking job for the exciting and glamorous life as a start-up CFO (ok maybe there were a few stressful parts to that story as well).