Bot traffic is internet traffic coming from automated software (bots) that’s designed to perform repetitive, mostly simple tasks. These bots can perform tasks around the clock, often much quicker than any human ever could.

Around half of all internet traffic comes from bots. While there are good bots that can be beneficial for your website, approximately 30% of all traffic comes from bad bots. These bots are designed to perform all sorts of malicious tasks, from scraping web content to stealing user accounts to scalping inventory.

What’s worse is that roughly two thirds of companies are completely unprotected against simple bot attacks. Even when bots are unsuccessful in executing their malicious objectives, they can still strain your web servers and hurt your website’s performance, potentially taking your website offline. Effective management of bot traffic is important for any business with an online presence.

Key Takeaways

• Bots account for approximately half of all internet traffic, with many of them being bad bots.
• While most bots are dangerous, there are some important good bots.
• Bad bots can lead to various issues, including data theft, server strain, and skewed analytics.
• Detecting and managing bot traffic is crucial for maintaining website performance and security.
• Effective bot management requires a combination of built-in tools and specialized solutions.

What are bots?

Bots are automated software applications designed to perform specific tasks on the internet. They can range from simple scripts to complex AI-driven programs. Bots are programmed to carry out repetitive tasks quickly and efficiently, often mimicking human behavior to interact with websites and web services. While some bots are good and necessary, like search engine bots and SEO crawlers, a significant portion of bot traffic comes from bots with bad intentions.

How do traffic bots work?

Traffic bots work by sending automated requests to websites, simulating human user behavior. They can be programmed to navigate through web pages, fill out forms, click on links, and even buy your products or services. Bots use various techniques to avoid detection, such as rotating IP addresses, human-like browsing patterns, and sophisticated user agents. Some advanced bots use machine learning algorithms to adapt their behavior based on website responses.

What types of bad bots exist?

Unlike good bots, bad bots don’t follow the rules in your robots.txt file. They tend to hide their identity and source, and often try to look like real visitors. But the main thing differentiating bad bots from good bots is in the types of tasks they perform: Hackers and fraudsters program bad bots to perform disruptive and even destructive tasks. They can cause permanent damage if left unchecked.

Web Scraping Bots

Web scraping bots steal content on your website to then publish or sell on other sites. For example, a web scraping bot can steal private price information about your products to release it to your competitors, so you lose your competitive advantage. This price scraping is pretty common for businesses where price is an important purchase decision factor, such as ticketing companies and travel agents.

Credential Stuffing Bots

These bots use stolen credentials (typically sourced from data breaches) to “stuff” known usernames and passwords into the login pages on other sites. The purpose is to gain access to user accounts. People tend to use the same username-password combination for all their accounts, so these attacks can have a high success rate.

Read more: Behind the scenes of a massively distributed credential stuffing attack

Spam Bots

These bots post spam content or send spam emails in bulk, often including links to fraudulent websites. We commonly see spam bots leaving comments on blogs, social media posts, and forums.

Ad Fraud Bots

Ad fraud bots click on pay-per-click (PPC) ads to generate extra revenue or skew the cost of the ad. As a result, the advertiser is charged with high advertising fees for a campaign that is not actually effective.

Denial of Service (DoS) Bots

In layer 7 DDoS attacks, bots make repeated requests to resource-hungry elements of a web application, such as large file downloads or form submissions. This causes slowdowns in performance, or even complete downtime.

Credit Card Fraud Bots

Credit card fraud bots make small transactions to find missing credit card information like CVV codes and expiry dates. They make these transactions until one goes through, which gives them the missing credit card information. Their activity often leads to chargebacks for e-commerce companies and can damage your business’ fraud score.

Gift Card Fraud Bots

Gift card fraud bots steal money from gift card accounts. They are popular because companies usually don’t protect gift cards as much as they protect credit cards. Still, gift card fraud will worsen your reputation with your customers and can eventually lead to significant revenue loss.

What are the consequences of bad bots?

If you don’t fully protect your business against bad bots, eventually they will cause some sort of serious problem. Although it’s hard to know exactly what problem you’ll face, some of the main consequences usually include:

• Higher bandwidth usage and increased server costs
• Skewed Google Analytics reports and other KPIs
• Lower conversion rates
• Poor website performance and user experience
• Increased strain on data centers
• Potential data breaches and security vulnerabilities

How to Detect Bots

Bot traffic must first be correctly identified before it can be managed. Here are a few things to look out for in your traffic and business metrics.

Increase in Traffic & Bounce Rate

An abnormal increase in traffic or unpredictable traffic spikes usually means a high number of bots coming to your site. Or a single bot coming to your site again and again. An increase in bounce rate indicates that the bot leaves without exploring more pages after it has fulfilled its task.

Page Load Speed

A dramatic dip in page load speed — especially if you haven’t made any significant changes to your website — is a telltale sign of bad bot traffic. Although bot traffic is not the only possible reason for slower site performance, it’s an indication that you should take a closer look at your other KPIs.

While one single bot is unlikely to make a significant impact on your site’s overall speed, many malicious activities involve a lot of bots entering a website at the same time, like in the event of Layer 7 DDoS attacks.

Abnormal Decrease in Bounce Rate

If your bounce rate dips to a suspiciously low level, it’s a strong indicator that web scraping bots are stealing your content or scalping your tickets. They’re essentially scanning a large number of pages, looking for opportunities to scrape content.

SEO Performance

This one is more difficult to measure right away, but when web scraping bots steal your content and publish it on other sites, it will eventually impact your website’s SERP ranking.

There’s a chance that your site might be outranked by the website that’s publishing your content. Your site could also get penalized by Google for duplication issues. Make sure to always set up canonical tags on every blog post so your article is always considered canonical even when your content is stolen.

Customer Complaints about Unavailable Goods

If your customers repeatedly complain that they’re unable to buy products from your website, you may be the victim of scalper bots. These bots are designed for ultra-fast online purchasing. They’re a cause of great frustration for real customers who are unable to beat them to the checkout page.

How to Stop Bot Traffic on Your Website

What should you do when you’ve determined that you have a bot problem? Well, you stop them. Although your main focus should be to stop bad bot traffic, you also need to manage traffic from good and verified bots. Not all good and verified bots may be useful for your site. While these bots won’t deliberately hurt your site, they could still strain your website’s performance with unnecessary traffic. Also, properly managing these good bots will help you differentiate them from bad bots.

Managing Good Bots

Good bots are open about their identity and are mostly willing to be managed. So managing their traffic should be fairly easy. There are two main approaches we can use:

Robots.txt

The main approach is to set up rules and policies in your robots.txt file. The basic principle is to let through the good bots that will benefit your site while blocking those that won’t.

Block & Allow Lists

If you have a bot management solution, the other approach is to set up a block list and/or allow list. For example, DataDome can set up an allow list for good bots that are allowed to roam our site. A good bot management solution should also let you manage good bot traffic with features such as rate limiting or timeboxing, so you can allow access on your own terms.

Managing Bad Bots

When it comes to managing and mitigating bad bots, there are several different approaches:

Invest in a Bot Management Solution

With bad bots getting better and better at imitating human behavior, an advanced bot management solution is the best way out. Nowadays, bots sometimes use AI and machine learning technologies to achieve their tasks and mask their identity. The best protection is an AI-based bot management solution like DataDome. Fight fire with fire.

DataDome performs real-time, behavioral-based bot detection to effectively identify even the most sophisticated bots, which can forge their user agent (UA) and rotate between hundreds if not thousands of perfectly clean IP addresses.

A lot of these bot management solutions are now fairly affordable and easy to use. If you are serious about your cybersecurity, investing in a proper bot detection and mitigation solution is a must.

CAPTCHA

A common approach to stop bot traffic is to use CAPTCHAs. But they’re not a one-size-fits-all answer to bot management. There are two reasons for this:

• Too many CAPTCHAs can ruin the user experience and increase your site’s bounce rate.
• Bots use CAPTCHA farm services to solve them.

Think of CAPTCHAs as a first line of defense. They are not the final answer to any bot problem. DataDome’s bot management solution comes with its own integrated DataDome CAPTCHA that takes, on average, less than 3 seconds to solve for humans.

Web Application Firewall (WAF)

Another common solution for stopping bot traffic is a WAF. Essentially a shield between a web page and your users. All traffic and resources first go through the WAF before they are sent to the user. If you’re technically minded, it’s like a reverse proxy server.

A WAF can be useful for protecting applications against the most common types of attacks. They may block a part of your unwanted bot traffic. But WAFs are designed for application protection, not bot detection. They are powerless against sophisticated bots that actively try to circumvent your security solutions.

IP-Based Management

It’s a good practice to block IP addresses that are obvious sources of malicious bots. Just be careful that you don’t block public IPs, since you could be blocking genuine users. And keep in mind that today’s bots typically use many different IP addresses, making IP-based protection increasingly ineffective.

Stricter Access Controls

It’s a good idea to implement stricter access controls on sensitive areas of your website. For example, areas intended for admins, or where users access your database. An extra security measure like multi-factor authentication can block bot traffic performing credential stuffing attacks or other malicious activities.

Conclusion

The most effective way to protect your business against bad bot traffic is through a specialized bot management solution. An AI-based one is preferred, because that’s the only way to stay ahead of bad bots using AI themselves. The best bot management solutions leverage machine learning to analyze visitor behavior and stop malicious bots before they even reach your network.

References

1. https://datadome.co/resources/datadome-global-bot-security-report-2024/
2. https://support.google.com/webmasters/answer/6062608