The challenge: Escalating bot threats leading to resource drain, skewed data, & fraudulent transactions
Etsy wanted to solve for the growing presence of “scrapers” which would gather information on publicly visible content by pretending to be shoppers. Mike Adler, the company’s Principal Systems Architect, recalls the early days when bot traffic was more of an inconvenience than a serious problem. “It started as something harmless we would notice, maybe a bit of annoyance, but we tolerated it,” he says. For many years, the bot activity was not critical enough to warrant interventions which would run the risk of bothering real users.
However, as Etsy grew into a larger marketplace, more bots showed interest in the platform. At the same time, Etsy was improving the customer experience with more and more machine learning, which can increase the amount of servers needed to serve pages. As a cost-conscious company, they wanted to stay in control of how resources were used.
Etsy analyzed their data and estimated that, if left unmanaged, scrapers could possibly account for about 1% of Etsy’s computing costs. “At our scale, 1% is big enough to justify making investments,” notes Mike.
As the bots grew into more of a nuisance, it took more work to prevent impacts. The scrapers might accidentally slow down one database and while it might not affect users, it could result in engineers being alerted. This was toilsome and caused annoying distractions.
The company was ready to try a more sophisticated approach to manage this growing threat.
