Web Scraping: Quantifying the Revenue Impact to Your Business
Web Scraping: Quantifying the Revenue Impact to Your Business

by PerimeterX Blog on May 21, 2020

While the debate rages on whether it is legal or not, web scraping, a classic business logic abuse threat is only getting bigger. It is commonly used by companies to keep pace with the competition. Scraping bots collect huge amounts of information, such as pricing data or product descriptions, and have to continuously perform this operation at scale to be valuable. Scraping bots technically are similar to web crawlers. Web crawlers, however, are typically good bots that help increase SEO rankings while scraping bots almost always create negative consequences for the websites being scraped. Understanding and measuring the economic impact of these consequences is a critical step in managing scraping bots.

The Scraping Marketplace

There are many publically available, affordable services that offer web scraping services. This indicates that there is a real market for such services. Some of the common providers are shown in the screenshots below.

Figure: Web scraping service providers

Also, there are several full-time and freelance job postings from e-commerce companies looking for web developers with scraping experience. The second listing shown below directly states that they want to find products that can be resold on various Amazon marketplaces for a profit of no less than 30%. In some cases, the listings also mention that the target website is protected by a commercial bot mitigation service – a clear indication of the importance of web scraping for such companies. Developers offering scraping services make use of CAPTCHA solving services through APIs and can automate fake account creations to get past the basic bot mitigation techniques on most websites.