An on-call IT worker receives an urgent call from the CEO requesting an MFA reset. The IT worker grants access – only to find out later that the voice they heard was not the CEO. The fraudulent access was used to initiate a ransomware attack and tens of millions of dollars in damages. 

An open-source contribution from a developer with three years of productive and helpful contributions includes an innocuous configuration script missed by other maintainers. The script installs a DLL that grants remote code access to developer machines and infects thousands of machines before discovery. 

Today, these scenarios require dedicated and organized attackers who methodically pick and research targets, exploiting public information to credibly impersonate or prioritize potential victims. These types of focused attacks are far more effective than older, “spray and pray” phishing schemes, but are expensive in terms of personnel and time. High-cost, high-reward, non-scalable. 

AI is About to Change This 

With the emergence of large language models (LLMs) and Generative AI, tasks that previously required significant investments in human capital and training are about to become completely automatable and turnkey. The same script kiddies who helped scale botnets, DDoS (distributed denial of service), and phishing attacks are about to gain access to social engineering as a service. 

Rather than a profit-motivated or state-sponsored hacking group picking one or two targets at a time, in a world of cheaply scalable, multi-channel attacks, this style of attack is going to radically expand. Instead of a single IT worker or executive assistant, attackers will now target all of them. Sophisticated attacks that become cheap will move “down-market,” to target multiple small and medium-sized businesses. For state actors pursuing strategic aims, why carefully infect one open-source project when bot contributors with synthetic identities can try to build trust in hundreds of thousands of them? 

When every phishing attack at scale is targeted and multichannel, every human interaction that isn’t face-to-face becomes a threat surface. When the cost gets cheap enough, the attack surface broadens horizontally and vertically, exposing far more companies, institutions, and individuals to attacks. We are already starting to see the beginnings of this trend, with attackers using AI to automate the identification of vulnerable companies and individuals.

Neither our institutions nor personal relationships are prepared for this.  The inherent trust of human-to-human interaction over voice calls, text messages, video conferences, social media, etc., is so ingrained into our business processes today that it will take years if not decades for our institutions to meaningfully respond to such a threat.  

As Terrifying as This Sounds, It is, In Fact, Significantly Worse  

The same AI that is being used today to generate fraudulent content and influence discussions on the internet is also capable of generating synthetic accounts that are increasingly indistinguishable from real, human accounts.  It is now becoming economical to completely automate the process of operating millions of accounts for years to emulate human behavior and build trust. 

Therefore, in a few years, looking at an account’s activity will no longer be a valid signal of whether that account is human or machine. Even if we stop naively trusting voice or video calls, the follow-up methods for filtering out bots or false identities we rely on today will also fail us.  

This is a far more immediate and substantial threat than quantum computing. AI botnets capable of credible social engineering attacks at scale are a certainty within months or years. Quantum-resistant encryption is meaningless if people and organizations can’t distinguish staff or customers from AI attackers.  

For identity security and social engineering, LLMs and Generative AI have effectively passed the Turing test. Therefore, any security paradigm relying on user behavior to distinguish humans from bots will fail. AIs are too good at credibly imitating and replaying human interactions online. The only answer is to provide people with a stronger form of identity, specifically a cryptographic solution without stored secrets. 

Avoiding stored secrets – eliminating the complexity and replay threats inherent to key management – is the only path forward to truly enable identity to enable authentication of you and not a proxy of you. Stored secrets will inevitably be compromised, so we must instead pick identity systems where the assertions of identity cannot be stolen or replayed. 

AI is forcing our hand. Unless and until we can empower humans to securely possess their own cryptographic keys, we face a world where AI advantages the attackers far more than the rest of us. 

This article was co-written by Cory Ondrejka, Creator of Second Life and a consumer tech executive who has held senior technical roles at the world’s largest companies.