Fighting Off Bots in 5 Milliseconds Flat
If we were living in a sci-fi movie world, the title might well be Rise of the Bots.
Today, it is said that more than half of web traffic is the work of bots (check out the Imperva Incapsula 2016 Bot Traffic Report). Software programmed to do automated tasks, bots can be good (chatbots for customer service, for instance), they can be bad (spambots), and they can be really, really bad (cybercrime botnets). “There’s a real black market for selling the illusion of human interaction on the web,” says White Ops co-founder/CTO Tamer Hassan, “and it’s worth so much money.”
In fact, in just the digital advertising space, bot-enabled fraud costs companies an estimated $6.5 billion a year. White Ops is the global leader in bot detection and human verification on the Internet. The company’s mission is to defend the open Internet and make everyone more secure by disrupting the profit centers of cybercrime. White Ops works globally with companies and industry groups that are dedicated to preventing malicious activity in advertising and analyzes over 50B events on a daily basis. “Instead of trying to make computers harder to break into, we make it less profitable to do so,” says co-founder/CEO Michael Tiffany. “If we disrupt the profit centers of cybercrime, then we make people more secure without them having to change their behavior. They just become less attractive targets.”
Tiffany and White Ops’s third co-founder and chief scientist, Dan Kaminsky, had been working on a solution for identifying bots in the financial industry when they met Hassan, a former Air Force search-and-rescue pilot who was investigating suspicions of fraud in the data of a company rebuilding an ad server.
As an experiment, the three decided to try embedding Kaminsky’s application in the ad server, which detected whether or not there was a human on the other end. “It was actually shocking how much fraud came into view that nobody knew about,” says Hassan. “There was always a suspicion that things like this happened, but that Google or Yahoo! or Microsoft caught it in their filters. That was probably the predominant assumption.”
After some research, Tiffany says, “It was clear that profits from ad fraud were far and away the biggest source of revenue for bot operators.” By either posing as legitimate publishers and serving ads that were never seen by humans, or by pretending to be digital media companies selling fake page-view traffic, bots were raking in huge profits. And that’s how White Ops decided to start their business.
“By taking the profits out of ad fraud, we could make it economically less viable for botnets overall,” says Tiffany, “so it became our most important priority.”
The team’s first, crucial task was to educate the advertising industry.
White Ops came out of stealth mode in October 2013 by sharing original research about the pervasiveness of the problem—and the economic repercussions. “I think that a lot of lightbulbs went off around this problem because, as Dan once put it, the parasite had grown so large that it started to endanger the host,” says Tiffany.
Thanks in part to this outreach, White Ops was able to launch an unprecedented global, coordinated shutdown of the high-profile Methbot operation in December 2016, which the company had discovered. (Incidentally, this was White Ops’ first interaction with Packet, where Kaminsky had retained a small fleet of dedicated servers for research and simulation.)
“The entire program was run out of a fixed set of IP addresses, so if everyone who needed to take action in the digital advertising ecosystem had the list of IP addresses that the Methbot operation was using, then their ability to monetize would end,” says Tiffany. Impressively, when White Ops released its findings and coordinated a shutdown response, the Methbot operation was destroyed within 24 hours.
Staying Ahead of the Bots
In their fight against botnets that are constantly evolving, Hassan says that “our model is to be two to three steps ahead of the bots.” Through an approach called threat modeling and a ‘white-hat’ hacker team that focuses on the problem, White Ops is continuously thinking of the best ways to game the ecosystem, and creating solutions before the bad guys can exploit them. “A good portion of what we do is considered zero-day, meaning that the moment somebody releases some form of attack, we already wrote a defense like a year ago.”
For enterprise users, the company introduced two products in 2015: FraudSensor, which is used to monitor and analyze ad impressions after they are served, and MediaGuard, which allows for pre-bid, probabilistic decisioning in real time, before ads are served. Mostly used in the programmatic ad space, MediaGuard “works with these really high-scale platforms that are running literally billions of events or ad impressions on a daily basis,” says Adam Morgenlender, White Ops’s VP of product.
"It was actually shocking how much fraud came into view that nobody knew about,” says Hassan. "There was always a suspicion that things like this happened, but that Google or Yahoo! or Microsoft caught it in their filters. That was probably the predominant assumption."
The Impact of Latency
To serve the programmatic ad space effectively, decisions need to be super fast. “We started off with a 2-minute decision window,” says Hassan. “Then, people like Yahoo! were telling us we needed to be at 5-milliseconds. And at a million a second.” Adds Tiffany: “That’s 5 milliseconds including the time it takes the data to travel across the network. Which means that we are processing data, analyzing it, providing a decision and sending the answer back in less time that it takes to light up the LCD on the display telling you we did it.”
“With Packet, we have access to raw bare metal, so we’re actually able to hit low level drivers on the machines themselves,” says Walker. With bare metal from Packet, the White Ops team was able to take their performance tuning to another level.
Says Morgenlender: “It’s just mind-blowing what we were able to get out of Packet: amazing performance and incredibly low latency to our partners, paired with the ability to scale up and down to meet demand.”
As a result of running Packet, White Ops was able to achieve the benefits of a colocation environment, but with full elasticity. This is a perfect fit for the fickle ad-tech industry, which experiences huge seasonal peaks in traffic - directly impacting White Ops’ need for costly infrastructure. For example, during the holiday season they need to be able to spin up a lot more resources to meet demand.
“Last year, our prevention numbers doubled in Q4, requiring us to quickly multiply our server count before scaling back down as traffic subsided in January,” says Walker. “Packet gave us this elasticity, but also enabled us to process more requests per second with fewer, less expensive servers. It was a double whammy of true savings, without compromising on performance.”
Enabling New Business
The White Ops team is also excited that the partnership with Packet has allowed the company to expand the coverage of locations it can support. Says Morgenlender: “We can now serve more customers in more locations without having to go to colocation. And as Packet expands its global footprint, we can get very close to an expanding group of potential customers.”
For Tiffany, the benefits of White Ops’s partnership with Packet can be boiled down to this. “Several of our customers, like AOL and Yahoo!, are some of the biggest internet properties around,” he says. “In order to protect and prevent ad fraud from happening on their platforms, we need to be able to reach the same scale as those companies, but at a cost basis that’s within reach for a security startup. And that, in my view, is what’s extraordinary about Packet. They’ve enabled us to succeed at our world-class mission on a global scale that matches internet behemoths while fitting our tech and economic needs. It blows my mind that this kind of thing is within reach for us.”
As for what the future holds for White Ops, Tiffany thinks the anti-bot technology the company has built has far-reaching applications. “I think that we can’t leave the advertising industry before the fight is won,” he says, “but we do have ambitions of providing this form of bot defense to the rest of the web, because so many different businesses are struggling with different forms of bot fraud and bot-enabled cybercrime. I see us ultimately growing into providing an extra level of authentication, almost authenticity, for web transactions of all kinds.”
In other words, the sequel is already being written.
Bots, you’ve been warned!
About White Ops - White Ops is the global leader in bot detection and human verification on the Internet. The company’s mission is to defend the open Internet and make everyone more secure by disrupting the profit centers of cybercrime. White Ops works globally with companies and industry groups that are dedicated to preventing malicious activity in advertising.