The Unseen Battle: How Bots and Automation Threaten the Web
The concept of the ‘Dead Internet Theory’ suggests a future where nearly all online content is generated and consumed by bots, with minimal human involvement. This might sound dystopian, but the surge in GenAI content and rising bot traffic means we’re nearly there.
Bots Are Evolving – And They’re Winning
New research from F5 Labs examined over 200 billion web and API traffic requests from businesses with bot controls in place. It highlights the increasingly advanced capabilities of bots and their operators who refuse to give up despite being continuously blocked. The report found that enterprises still saw an average of 10.2% of their traffic originating from bots. For businesses without such defenses, bot-related traffic would be significantly higher, but since bots attempt to appear as human as possible, many businesses have no way to quantify the level of malicious automation they face.
Hospitality and Entertainment Industries Hit Hard
Bot traffic fluctuates wildly across industries, but the most afflicted with bots was hospitality, with 45% of all web traffic in this sector coming from unwanted sources of automation. Typically, bots attacking this sector are scraping prices and availability data to fuel competitors looking to undercut pricing. Looking at APIs delivering content for mobile apps, the entertainment industry appeared worst hit, with 23% of all unauthorized traffic originating from bots.
Malicious Bots: Scrapers, Resellers, and Credential Stuffers
Bots vary in form and capability depending on their target. Some of the most concerning include content scrapers, reseller bots and credential stuffing bots.
The seemingly infinite appetite of gen-AI large language models (LLMs) is partly to blame for the explosion of content scrapers, with an average of 50% of all web page requests for content pages (e.g., displaying product information, prices, etc.) coming from bots.
Reseller bots, also known as scalper bots, purchase high-demand items in bulk before real users can, reselling them at inflated prices. Fans of Taylor Swift or new tech like GPUs know the pain of instant sellouts. Tracking the use of ‘add-to-cart’ webpages and APIs, the F5 Labs report found up to 21.5% of all purchases were attempted by bots.
Perhaps the most dangerous of all bots are the credential stuffers, otherwise known as account takeover bots. Credential stuffing remains one of the biggest cybersecurity threats across all industries, relying on databases of stolen usernames and passwords, testing them against various websites to gain access, ultimately leading to fraud and identity theft. Some of the worst-hit industries included technology, with more than a third of all login attempts coming from credential stuffing bots. The retail industry also suffered high numbers, with almost 26% of logins originating from unauthorized bot login attempts. For mobile APIs, entertainment saw 25% of its login traffic linked to credential stuffing, and e-commerce had 24% of login attempts coming from bots attempting account takeovers.
Advanced Bots are Harder to Detect Than Ever
Bot creators rarely make their bots any more capable than they need to be. Indeed, many bots were found to be relatively unsophisticated, with 90.5% of bots scraping website content classified as ‘basic’, meaning they had limited capabilities. These bots often identify themselves as command-line tools or applications, making them straightforward to identify and block.
However, advanced bots present a greater challenge. These bots mimic human interactions, executing JavaScript, moving the mouse in an erratic pattern, and even simulating human browsing behavior to evade detection. This is particularly concerning in industries where reputation and trustworthiness are crucial. For example, 91% of bots targeting online rating and review systems were categorized as ‘advanced’, showing how businesses use automation to generate fake positive reviews.
Residential Proxies: The New Front in Bot Defense Evasion
The F5 Labs report also uncovers the thriving market of residential IP proxy networks. A common defense against bot activity is IP-based blocking, which includes rate-limiting, using IP reputation databases, blocking connections from cloud hosting services, TOR exit nodes, and VPNs, and implementing geolocation-based restrictions. However, these strategies are becoming less effective as bot operators adapt. To bypass IP-based controls, attackers rely on vast pools of residential proxies, which route bot traffic through legitimate home broadband and mobile networks. These proxies make bot traffic look human. Many proxy networks claim to offer over 100 million unique IPs, but real-world testing suggests that only a fraction are in active use.
Are We Already Living in the Dead Internet?
The rise of bots is not a passing trend; it is a fundamental shift in how the web operates. With bots employing advanced techniques to appear human and masking connections behind residential IPs, organizations that rely on pure IP-based controls will find their efforts increasingly ineffective. Without strong security measures, the internet may inch closer to becoming the automated, bot-driven landscape envisioned by the Dead Internet Theory, where real users are just an afterthought.
