In a Social Engineering Showdown: AI Takes Red Teams to the Mat
In a revelation that shouldn’t surprise, but still should alarm security professionals, AI has gotten much more proficient in social engineering. Back in the day (that would be two years ago), AI was 31% less effective than human beings in creating simulated phishing campaigns.
But now, new research from Hoxhunt suggests that the game-changing technology’s phishing performance against elite human red teams has improved by 55%. While security teams have anticipated the improvements AI will bring to their efforts — the ability to identify patterns, predict attack vectors, simulate complex scenarios and scale up — threat actors have been in the shadows, busily developing their own generative AI tools, something that Hoxhunt has been tracking closely these last two years.
“Humans are uniquely creative and adaptable, which matters when you’re attacking and adapting to the defenses and responses of a human target. AI doesn’t have the same context to adapt,” says Casey Ellis, Bugcrowd founder. “That said, AI attack agents can operate at scale and never need to sleep, so if the goal is breadth versus depth, AI will tend to perform better.”
With a nod to The Terminator, Hoxhunt says the Skynet Moment for AI social engineering came in March when the company’s AI agents created simulated phishing campaigns against global users that were more effective than those created by human red teams. In 2023, AI was 31% less effective than humans before zipping past those red teams in 2025. By the numbers, the company found that:
- Just .7-4.7% of phish that bypass email filters were written by AI in 2024.
- 49% increase in phishing attacks that bypass email filters since 2022.
Those findings seem even more dire when you consider that Slashnext reported a 4,151% increase in phishing volume after ChatGPT’s debut in 2022 and underscore the duality of AI.
“Like any powerful tool, AI can be used for good or for evil,” says Pyry Åvist, Hoxhunt CTO and co-founder.
“AI systems are susceptible to adversarial attacks, where malicious actors might deceive them by exploiting their algorithms,” says Jason Soroko, senior fellow at Sectigo.
“Ethical considerations arising from the use of AI for offensive cybersecurity include the potential for AI to be used maliciously if it falls into the wrong hands,” agrees Stephen Kowski, Field CTO at SlashNext Email Security. “There’s also the risk of unintended consequences when AI systems make autonomous decisions.”
Privacy Laws and Ethical Standards
Using AI offensively can raise legal and regulatory hackles related to privacy laws and ethical standards, Soroko adds, as well as creating a dependency risk. “Over-reliance on AI could diminish human expertise and intuition within cybersecurity teams.”
But that doesn’t mean bad actors will win the day or get the best of cyberdefenders. Instead, security teams could and should turn the tables on them.
“The same capabilities that make AI an effective phishing engine can — and must — be used to defend against it,” says Avist. With an emphasis on “must.”
“In today’s landscape, AI is a necessary tool to keep pace with evolving threats, including those posed by attackers who are increasingly leveraging AI themselves,” says Amit Zimerman, co-founder and chief product officer at Oasis Security. “As adversaries become more sophisticated, organizations must adopt AI-driven offensive cybersecurity to stay ahead, making AI not just a convenience, but a critical asset for maintaining a competitive edge in security.”
To get there, though, security teams still must clear a significant, persistent hurdle: Developing both AI’s security acumen and guardrails around security and governance of AI after the fact, while it’s on the loose and moving fast through business.
And organizations are overwhelmed by the sheer pace of changes to AI and the regulatory bog that they must wade through to make sense of everything. “In our discussions with customers, it is evident that they are overwhelmed on how to prioritize and tackle the issues — there’s a lot that needs to be done,” says Acuvity CEO and Co-founder Satyam Sinha.
Transparency is Crucial
It seems that tried and true basics are a good place to start. “Ensuring transparency, accountability and responsible use of AI in offensive cybersecurity is crucial,” Kowski.
As with any aspect of tech and security, keeping AI models “up-to-date with the latest threat intelligence and attack techniques is also crucial,” he says. “Balancing AI capabilities with human expertise remains a key challenge.”
That latter part is telling and offers a clue to maximizing AI’s effectiveness while reducing risk. Don’t take the people out of the equation, lean into them. “The combination of AI technology with human expertise creates the most effective approach to offensive cybersecurity,” says Kowski. “Human oversight is essential because AI, while powerful, can’t fully replicate human intuition and contextual understanding.”
Vendors have an important role to play as well, in giving defenders the highest odds of besting bad actors and spurning their nefarious use of AI. “The gap in confidence and understanding of AI creates a massive opportunity for AI native security products to be created, which can ease this gap,” says Sinha.
The AI landscape is no place for shrinking violets and hesitant heroes. Security defenders everywhere should take the Hoxhunt findings not only as a loud clanging alarm but as a call for action. And understand that bad actors are doing the same.
