One of the fun things we have going on in our Labs is our PacketBot project. A PacketBot is basically a Raspberry Pi configured with some network monitoring scripts which send us network performance data. We ship the 'bots around to various home and work locations to get an idea what the Packet network looks like at various locations and service providers. We then push all this network performance data to a centralized OpenTSDB data store and visualize it with Grafana, a graphing frontend for time series data stores.
We have some other big plans for our beloved PacketBots: they'll make regular appearances in our client portal, be able to alert you when something is wrong, and maybe even help with some two-factor authentication to keep things nice and secure.
But for now, sending us all this delicious network performance data is job #1.
Currently, the PacketBots are all safely housed with Packet staff and friends, so they are pretty secure. But long term, we’ll also be letting clients and other folks adopt them and take them home, which means we’ll need a secure way to remotely troubleshoot and maintain them. We also want to be able to push changes to our scripts and add functionality as we need to, so we did a little technology round-up to see what the best fit for managing our Bots was.
We deploy all of the Packet software on CoreOS and Docker, so are familiar with that, but unfortunately the CoreOS support for the Raspberry Pi platform is just not ready for prime time yet, so this was more or less off the table. We did tinker around with using Docker on Raspbian, the Debian port to the Raspberry Pi platform, but ran into enough “weird problems” that we decided that was not a great option either - especially since we’d still need to manage the base Rasbian operating system anyway.
We are also very familiar with Puppet, which we use to manage hundreds of servers and infrastructure devices; so we decided to investigate that, Chef (which we also have some experience with), and Salt (which we didn’t).
One of the things about Raspberry Pi, is that when you are used to working with high performance server hardware, you realize how sloooooooow the Pi's are. That ARM processor does a valiant job, but it is pretty easy to max the little guy out. We want our Bots to be happy, healthy and well rested, so after looking into Salt and seeing its more lightweight approach to distributed configuration management, as well as the ability to natively execute commands (similar to DSH or Mcollective), we decided to give it a shot.
Verdict: we love it!
For our simple PacketBot needs, Puppet or Chef would have been too much of a performance overhead and unnecessarily complex in configuration and setup. Both tools have their uses, but for this project, Salt really ended up fitting the bill.
So, how does it work?
First, we setup a “master” Salt server, which the Bots talk to over a vpn. Then, when we setup a new Bot, configure the Salt client on the Raspberry Pi so it knows where the master is, and when it runs for the first time, it checks into the master, and they exchange keys. The new Bot isn’t accepted by the Salt master initially and requires us to “accept” the new key first. Here you can see a listing of the salt keys on the master, as well as a test ping of one of the Bots.
From here on out we can now manage the Bot with Salt, including package management, service configuration, and remote command execution. We’ll do another post soon with a more in-depth review of how we are using Salt on the PacketBot network, so stay tuned.
We’re really excited about everything we have cooking over here, the PacketBots included. If you would like to adopt one for yourself, send us an email at firstname.lastname@example.org for an application.