What is a bot? A Guide to Good Bots, Bad Bots, and Bot Protection

For better or for worse, bots have become an integral part of the internet. Some bots, like the Googlebot, are essential for the internet to function properly. But most bots are not friendly. Most bots are dangerous and create significant cybersecurity threats for businesses and individuals.

According to our 2024 Global Bot Security Report, nearly 65% of websites are unprotected against even basic bot attacks. This highlights the need to understand what bots are and how you can address this important problem.

How do bots work?

A bot is a software application that automatically performs certain tasks quickly and at scale. It is a tool that can be used for good or bad purposes. Good bots are integral to our daily online lives, while bad bots can seriously damage your business if you don’t properly protect yourself against them.

The Two Different Types of Bots

The easiest way to classify bots is by their intent: do they have good intentions or not? Are they good or bad? More technically: Do they follow your robots.txt file or not? Good bots will follow the instructions in your robots.txt file, while malicious bots won’t. Apart from this broad classification, bots vary widely in complexity. They can be a simple chatbot, a few lines of code meant to automate a repetitive task, or multiple scripts working together to mimic the behavior of a human.

What is a good bot?

A good bot is a bot that performs a helpful or useful task for your company or website visitors. It is not built with bad intentions. Most of the time, it does not damage or worsen the user experience of the places it crawls.

A good bot is usually built by a reputable company like a search engine or social media platform. It respects the webmaster’s rules that regulate how often web crawlers should index a website. These rules are usually defined in a website’s robots.txt file. A good bot should be programmed to look for that file, read it, and follow its rules before it does anything else.

Examples of good bots include:

• Search engine bots like the Googlebot
• Social network bots like Facebook Crawler
• Aggregator bots like the Feedly Fetcher
• Marketing bots like the SEMrush bot
• Site monitoring bots like Uptimebot
• Voice engine bots like Alexa’s crawler

What is a bad bot?

A bad bot is programmed to perform a task that will hurt your company or website visitors. It is built with bad intentions by cybercriminals, fraudsters, or even your competitors, and will directly or indirectly worsen the user experience of the places it crawls. A bad bot doesn’t read or ignores the rules in your robots.txt file.

According to our Bot Report, malicious bots are becoming increasingly sophisticated. The report revealed that advanced bots were detected less than 5% of the time, indicating a significant security challenge.

Common malicious bot types include:

• Scraper bots that steal your content, prices, and publicly available information
• Credential stuffing bots that attempt to take over user accounts
• Spam bots that spread unwanted content on your web pages or elsewhere
• Scalping bots that hoard your inventory and tickets to resell them

How to Identify Bot Traffic

Although bots vary widely in their intent and complexity, there are usually a few common signs that indicate your mobile app, website, or API may be suffering from bot traffic:

Unusual Traffic Patterns

Sudden, unexplained spikes in internet traffic often indicate bot activity and accompanying DDoS attacks. Pay particular attention to traffic surges during unusual hours, when your target market would typically not be active.

For instance, if your business primarily serves North American customers but you’re seeing significant traffic at 3 AM EST, this could signal bot activity.

Response Metrics

Bot attacks can strain your servers in distinctive ways. Monitor your bounce rate carefully. If it suddenly increases and you haven’t made any major changes to your website, this could indicate bot traffic that’s visiting pages and immediately leaving.

Additionally, watch for patterns in server performance. Bots often make requests in ways that differ from human users, such as maintaining perfectly consistent intervals between actions or generating an unusually high number of requests per session.

Technical Indicators

Keep an eye on suspicious IP addresses, especially those from data centers or regions known for malicious activity. Our Bot Security Report shows that modern bots have become adept at rotating through residential IP addresses, making them harder to detect through IP analysis alone.

Additionally, monitor user agent strings and browser fingerprints. While sophisticated bots can spoof these, inconsistencies in these technical markers often reveal simpler bot activity and malware.

Basic Protection Measures to Prevent Bad Bots

1. Exclude bots from Google Analytics. While Google Analytics offers basic bot filtering, follow those recommendations to exclude more bot traffic but it won’t catch sophisticated bots searching for vulnerabilities in your defenses.
2. Use Captchas. Though traditionally effective, modern captcha bots have become much better at bypassing CAPTCHAs, making them more of an annoyance for humans than a deterrent for bots.
3. Implement a WAF. Web Application Firewalls can block known threats, but struggle with modern bots that are part of a botnet or rotate through residential IPs.
4. Integrate MFA. Multi-factor authentication helps protect login pages, but user adoption remains a challenge.

Advanced Bot Protection to Protect Your Business

A dedicated bot management solution is the easiest, most cost-effective, and best way to protect your business against bad bots. Independent research firms consider bot management an increasingly mature industry that’s beneficial for companies to invest in. Bot management solutions have become required cybersecurity software to protect your websites, mobile apps, and APIs.

Key features to look for in a bot protection solution:

• Real-time detection and blocking (under 2ms)
• Protection across all endpoints (cart pages, login pages, and APIs)
• Machine learning-based behavioral analysis
• Ability to adapt to new threats
• Easy deployment across different tech stacks

DataDome is real-time bot management software that protects your business against all bad bot attacks. It detects and blocks bad bots in less than 2ms, even if those bots rotate through thousands of IPs to make them harder to detect.

Additionally, DataDome understands that every company has a different tech stack. Your website architecture, no matter how complex, shouldn’t hold you back from investing in the right bot management solution. DataDome works on any web infrastructure and can be deployed in minutes. There’s no complex setup, although you can customize DataDome to your heart’s desire if you so please.

The bad bot landscape is continuously evolving. That’s why, when the DataDome algorithm detects a new bot threat on any of the properties it protects, it will spread that knowledge to all DataDome customers in less than 50ms.

Conclusion

Traditional bot defense methods have become ineffective. To effectively protect your websites, mobile apps, and APIs against both basic and advanced bot attacks, you need an advanced bot management solution.

Our BotTester tool can give you a peek into the basic bots reaching your websites, apps, and/or APIs. If you’re ready to learn how DataDome can keep your business safe in the most terrifying of threat landscapes, book a demo today.