How Faster Builds Could Have Saved us 21 Days in 2015

January 08th, 2016

Aaron Welch

Aaron Welch

SVP, Product
In 2015 we lived the microservices + CI dream, pumping out over 3000 builds via shippable.  There was just one problem: the builds were SLOW.  Here's how I reduced build time by 69% and saved our team 21 days of not waiting for builds in 2016. 

In July 2014, when we first started sketching out the Packet platform, we quickly recognized the value of a microservices based architecture. Like many other startups circa 2015, we also embraced the container revolution and the value of CI (continuous integration) as a cornerstone of proper development workflow. Many of us had done it the "old" way for many years, so we knew it would be an adjustment, but with Packet we were determined to look through the windshield and not in the rear view mirror.

We are really fortunate to have a team that is invested in this approach, and are willing to spend the time to "integrate all the things" - no small task to stitch together the many many pieces of the puzzle. When it came to selecting a tool for our build tests we happily landed with shippable.  And with that, we were off to the races (so to speak...but more on that next!).

Fast forward to December, 2015. I was sitting at my desk, looking at an astounding number: over the past twelve months the devs had triggered over 3000 shippable builds. That’s awesome right? I mean, our team crushed it - living the CI dream to the tune of like 8 builds a day, 7 days a week for a year!!

There was just one problem that these numbers didn’t reveal. The builds were SLOW. Like watching paint dry slow. Glacial. Like waiting for Luke to come back from his self-imposed weird exile slow. It was excruciating, and horrible for keeping the pace of things moving, especially for small fixes and troubleshooting.

Any little change that needed to go from local dev to our lab environment required waiting on a build, and felt like it was taking forever. Even our little services took almost 10 minutes, and the worst case was about 25 minutes for our core API to build. Running the numbers in my head, I estimated we’d spent about 45,000 minutes this past year waiting for builds. Agh! 

No wonder the single most common gripe in the Packet #productdev Slack channel is about waiting for builds.

Lucky for us, Shippable recently released a preview of their “Single Tenant CI” build plan, which lets you run builds on your own infrastructure. As an infrastructure company, we just happened to have a few servers lying around. So as a year-end gift to the team (and myself), I decided to do something about it.  Game on!

(To be fair, I had already laid the groundwork for this impromptu hackathon by conducting a POC of buildkite the week before. I was super impressed, and while I still have plans for a benchmark shootout, I decided to see what a bit of Packet bare metal would do for improving our shippable build times in the short term.)

To get started I spun up two machines: our new Type 0 as well as a Type 1. About 7 minutes later I was able to load up the shippable agent on each machine. I was curious what the performance difference would be, so initially I just registered the Type 0 in shippable, and kicked off a few builds, mixing some of our more lightweight ones with some of our heavier ones. I then did the same with the Type 1 machine.

On both server types it appears that the initial few builds were still sluggish, probably due to downloading all the required libraries and requirements, but then sped up significantly on subsequent builds.

While the Type 0 performed well, cutting build times roughly in half, there was no doubt that the much beefier Type 1 was going to make a better stocking stuffer for the team. So I doubled down on the larger machine and ran all the builds a few times to get some data.

The result: build times dropped from 25+ minutes for our API to less than 7 minutes, and many of our other services went from 10+ minutes under a minute. Talk about getting bugs stuck in your teeth fast!

So, by spending an hour or so and $200 I was able to reduce build times by 69%. If we do a similar number of builds next year - and my hunch from looking at the roadmap is that we will increase that - it means that this small time investment and some Packet hardware will save us something like 504 hours of time NOT waiting for builds to finish. Woot! How else can you grab yourself an extra three weeks of productive development time. What is that worth to your team?   Thousands, if not tens of thousands, of dollars I’m sure.

Not bad ROI (especially since the shippable single tenant plans are free while in preview!). So sign up for some bare metal (our Type 0 works quite well for this at just $.05 per hour) and bask in the glory as your team showers you with accolades and high fives for making their quality of life demonstrably better with no extra work on their end!

Still have questions? We're here to help!

Contact Us

Call: 212-993-9785


Follow: @packethost