Report Finds LLMs Are Prone to Be Exploited by Phishing Campaigns
A report published this week by Netcraft, a provider of a platform for combating phishing attacks, finds that large language models (LLMs) might not be a reliable source when it comes to identifying where to log in to various websites.
Out of 131 hostnames for 50 well-known brands identified by LLMs in response to a natural language query, just over a third (34%) turned out to be owned by some other entity, with 29% of the domains being unregistered, parked, or otherwise inactive.
Robert Duncan, vice president of product strategy for Netcraft, said that creates an opportunity for cybercriminals to then set up fake websites that might be used to lure unsuspecting end users into providing their credentials for logging into legitimate websites.
The report also notes that this issue goes well beyond a theoretical construct. When an LLM was asked to surface the login site for Wells Fargo, the answer provided turned out to be a phishing site that had been previously set up to steal customer credentials.
Many end users are now relying on LLMs rather than search engines to launch queries, but the responses being provided may not be as trustworthy, noted Duncan. Major brands have spent decades making sure their domain ranks highest by investing in various search engine optimization (SEO) technologies, he added. LLMs, however, are surfacing results based on samples of data randomly sampled from across the web.
The Netcraft report also notes that various threat actors have already generated more than 17,000 AI-written GitBook phishing pages targeting crypto users, many of them styled as legitimate product documentation. Those efforts are now being extended to create phishing campaigns aimed at the travel industry.
Additionally, malware is being distributed via “cracked software” blogs, tutorials and discussion posts that might be surfaced using AI search tools.
In another campaign, Netcraft uncovered an effort to poison AI coding assistants. The threat actor created a fake application programming interface (API), SolanaApis, designed to impersonate a legitimate Solana blockchain interface. Developers who unknowingly included this API in their projects were routing transactions directly to the attacker’s wallet. The attacker didn’t just publish the code. They launched blog tutorials, forums and dozens of GitHub repos to promote it. Multiple fake GitHub accounts shared a project called Moonshot-Volume-Bot, seeded across accounts with rich bios, profile images, social media accounts and credible coding activity. These accounts were specifically crafted to be indexed by AI training pipelines, noted Duncan.
Netcraft has discovered at least five victims who copied this malicious code into their own public projects. Those poisoned repositories are now feeding back into the training loop to attack the AI supply chain attack itself, added Duncan.
It’s not clear how well these efforts to poison AI search results are working or may simply be overwhelmed as AI search tools become better trained. However, the one thing that is clear is that AI search tools are not as reliable as most end users believe, so the challenge now is educating end users about the actual level of potential risk these tools actually represent.
