Carsforsale Leverages Real-Time Scraper Bot Protection With DataDome

Carsforsale.com is a software platform for automotive dealerships, which helps more than 22,000 auto dealers connect with car buyers via a range of sales and marketing tools. When their previous bot protection solution was sunsetted, the engineering team vetted multiple providers and selected DataDome. Today, Carsforsale enjoys clean traffic to their website and the websites they provide for dealers, and the company’s engineers can focus their energy on building excellent automotive software.

The Problem: Scrapers Targeting Content and Draining Resources

Like all marketplaces, Carsforsale.com and the custom websites they provide for auto dealers are attractive targets for bots trying to scrape and exploit online content.

“We like to focus our engineering efforts in our domain of expertise, which is not fighting bots,” says Nick Johnson, Software Engineering Manager at Carsforsale. “There have always been bots around. We knew and could see bad actors coming into our systems, but staying on top of those and mitigating all the new ones coming in was just too big of a challenge. It was also taking away from our core competencies, which is to provide excellent software for our dealerships and the consumers that visit our site.”

Carsforsale owns its hardware, and even major scraper bot attacks never really threatened to take down its sites. Nonetheless, the engineering team didn’t appreciate the scraping attempts nor the resource drain they represented.

“At times, we would definitely notice performance issues,” Nick observes. “Bots would crawl our sites in ways that normal users can’t, and as a result, we had to scale a bit larger to make sure we could handle that influx.”

To mitigate this unwanted bot traffic, the team used an on-premise product for some time. Then, their solution was sunsetted and replaced with a cloud-based version.

“Since we had to go the alternate route of hitting the vendor’s servers anyway, we decided to take a look at what was out there in the market, and maybe find a solution that would better fit our needs,” Nick explains.

The Solution: Cost-Effective Real-Time Bot Detection

The team initiated conversations with multiple providers, but many were immediately ruled out due to prohibitive costs.

To choose from the shortlist of solutions that proved efficient at identifying and blocking bots, and which remained within the range of what Carsforsale was ready to spend, the most important selection criteria were user-friendliness and latency.

“We wanted to take usability and reporting a step above what the previous solution had been offering,” Nick notes. “We didn’t have a lot of visibility into what they were actually blocking and why, and we wanted better insights into the decisions that were being made.”

Being used to an on-premise product, the team also wanted to ensure that the new solution didn’t negatively impact performance. With its 25 global edge points of presence, DataDome passed the test with flying colors.

“We didn’t know what to expect in terms of performance impact, so we ran a test to check the round trip time in DataDome’s system to determine if we had a bot on our hands or not. The results were very satisfying,” Nick confirms.

The Results: Real-Time Protection and Detailed Reports

Once the engineering team had decided to entrust Carsforsale’s bot protection to DataDome, they wanted to get it up and running fast.

“The implementation was very quick,” Nick recalls. “We made a few changes in HAProxy, and put a JavaScript snippet in the front end product. It took only about a day for a couple of people, and we really appreciated the attention from the DataDome onboarding team. They never left us on our own or waited for us to ask questions. They were very engaged in ensuring that we understood how to use the product, and that we were seeing the metrics we wanted to see. It really felt like a partnership, rather than just a business transaction.”

Since the former solution hadn’t provided very detailed reports, the team was surprised at the amount and variety of bots—around half of the website’s total traffic—they discovered in the DataDome dashboard.

“I appreciate the granularity that we get,” says Nick. “We can take the 10,000 foot view and just see the percentage of bad bots, good bots and commercial bots, but we can also dig into those categories and get really granular, down to a single request. It helps us understand the reasoning behind blocking decisions and how bots are classified, and gives us confidence that DataDome is making the right decisions for our users.”

The dashboard also makes it easy to add custom rules, which the Carsforsale team primarily uses to allowlist friendly commercial bots. For example, SEO partners are allowed to scrape the website for the purpose of their analyses. But overall, the team now spends very little time and energy on bot-related problems.

“As I said initially, we don’t want to be bot experts. Partnering with a company like DataDome allows us to focus our energy on building automotive solutions,” Nick concludes. “Someone on the team will regularly consult the dashboard just to check on a few things, but for the most part, DataDome runs on autopilot. It gives us a great deal of peace of mind.”