Red-teaming agentic AI: New guide lays out key concerns for AppSec

A new guide published by the Cloud Security Alliance (CSA) gives red teams some useful guidance on how to go about securing agentic AI systems. Red-teaming for agentic AI requires a specialized approach because the planning, reasoning, tool utilization, and autonomous capabilities of those systems create attack surfaces and failure modes that extend far beyond those present in standard large language models (LLMs) or generative AI models, the CSA guide highlights.

While both agentic and non-agentic LLM systems exhibit non-determinism and complexity, the guide explains, it is the persistent, decision-making autonomy of agentic AI that demands a shift beyond traditional red teaming in how the agents are evaluated and secured. These unique challenges underscore the urgent need for industry-specific guidance on effective red-teaming agentic AI applications, the guide adds.

Ken Huang, co-chair of the CSA’s AI Safety Working Group and lead author of the 62-page guide, said that the attack surface of an agentic system is significantly more complex and expanded to include not just the model’s input but also its control system, goals, knowledge base, and interactions with other agents.

“The autonomous nature of agentic AI requires a new security evaluation approach because these systems can plan, reason, and act on their own, interacting with external tools and APIs in ways traditional red teaming doesn’t cover,” Huang said. “Unpredictable and emergent behaviors are a core challenge of agentic systems, as their non-deterministic nature means identical inputs can lead to different, and potentially harmful, actions that standard testing would miss.”

Huang also noted that the risk of cascading failures is unique to these interconnected systems, where a single compromised agent can propagate an attack across every tool and system it has access to, creating a massive blast radius.

“Agentic AI behaves less like a program and more like an autonomous operator, requiring a new red-teaming framework that can test its complex, interactive, and unpredictable nature.”
—Ken Huang

Here are the key considerations outlined in the new CSA guide on red-teaming agentic AI.

[ Get White Paper: How the Rise of AI Will Impact Software Supply Chain Security ]

Risks go well beyond prompt injection

Agentic AI systems behave autonomously, and that means that they introduce risks far beyond prompt injection, said Stuart McClure, CEO of Qwiet AI. Traditional red teaming falls short, he said.

“We need a framework for dynamic, goal-driven systems that evolve in real-world environments.”
—Stuart McClure

Rosario Mastrogiacomo, vice president of strategy and solutions engineering at Sphere Technology Solutions, said that agentic AI creates entirely new categories of identity-related risks — particularly around autonomous privilege escalation, credential misuse, and unauthorized identity propagation.

“Without a specialized guide, security teams risk missing these unique identity vulnerabilities, leaving critical access points unprotected.”
—Rosario Mastrogiacomo

The CSA guide pushes red teaming beyond isolated tests and toward a systematic, repeatable approach tailored to agentic AI, said Melody (MJ) Kaufmann, an author and instructor at O’Reilly Media.

“By organizing risks into 12 threat categories, security teams can clearly and structurally uncover vulnerabilities across the full lifecycle of autonomous agents.”
—Melody (MJ) Kaufmann

Those categories include agent authorization and control hijacking, checker-out-of-the-loop, agent critical system interaction, goal and instruction manipulation, agent hallucination exploitation, agent impact chain and blast radius, agent knowledge base poisoning, agent memory and context manipulation, multi-agent exploitation, resource and service exhaustion, supply chain and dependency attacks, and agent untraceability.

Mastrogiacomo said the CSA guide comprehensively covers major threat categories but could further emphasize identity-specific threats, such as unauthorized AI-driven role assumption or cross-domain identity federation vulnerabilities. “Explicitly including these would strengthen coverage of AI identity governance,” he said.

While the list is extensive, the space is still emerging, Kaufmann added. “New risks could arise from novel orchestration models, cross-modal agents, or real-world tool integrations, such as robotics, finance, or biotech. But the guide’s format allows for ongoing expansion,” she said.

McClure agreed that the agentic security field is still a work in progress.

“Future gaps may emerge in RAG exploitation or deeply integrated model-agent hybrids, among other areas.”
—Stuart McClure

Software supply chain threat is legit

“Agent authorization and control can be hijacked directly, allowing an attacker to manipulate the agent’s decision-making process, exploit its permissions, or abuse its dynamic role assignments, which are functions a standard LLM does not possess,” the CSA’s Huang explained.

Huang noted that a recent Microsoft Copilot security flaw discovered by security firm AIM is such a case. Microsoft 365 Copilot had a critical vulnerability called EchoLeak that let attackers access sensitive data simply by sending an email with hidden prompts, which Copilot would process automatically without user interaction.

This flaw, rooted in how Copilot’s AI handled instructions embedded in messages, allowed unauthorized access to files, emails, and chats, and was traced to a fundamental design issue in LLM-based AI agents. “Microsoft has patched the issue, but experts warn that similar risks exist for other autonomous AI systems,” Huang said.

Another threat category — interactions with critical systems — can present a severe failure mode, where an agent connected to physical robotics or industrial controls could be manipulated to cause dangerous real-world outcomes, not just generate a flawed text response.

Avenues for collusion and deception can be opened up by exploiting multi-agent orchestration, he continued, where attackers can manipulate the trust protocols between agents, causing them to work together maliciously or tricking one agent into misusing its authority. He also stressed that software supply chains can also be threatened by compromised agents, because the agent’s impact chain creates a large blast radius, and since a compromise is not isolated, can propagate and turn a single breach into a system-wide catastrophe, he said.

Agentic AI does more than add complexity to the software supply chain, O’Reilly’s Kaufmann noted.

“It can accelerate and automate bad decisions, creating fast-moving supply chain compromises that evade human detection.”
—Melody (MJ) Kaufmann

Compromised agents can propagate malicious code, abuse CI/CD permissions, or tamper with plug-in and dependency chains,” Qwiet AI’s McClure added. “Since agents often execute or orchestrate builds, they can become a vector for deep, invisible compromise,” he said.

Huang also noted the lack of agent traceability can be exploited to obscure malicious activity, as an agent can perform a long, complex chain of actions using multiple tools, making it nearly impossible to trace an attack back to its origin.

CSA guide is a living document

Overall, the CSA guide provides a solid foundation, but future versions should more explicitly integrate robust identity governance frameworks tailored to autonomous agents, said Mastrogiacomo. “Clearly emphasizing identity hygiene and lifecycle management as central to red-teaming efforts will further strengthen organizational resilience,” he said.

McClure said that the CSA guide is a blueprint for evolving from static checks to dynamic simulations. But that doesn’t replace static or dynamic analysis, he stressed. “It amplifies its importance,” he said.

“Before agents act, ensure the foundational code they rely on is secure and resilient. You must secure the foundation of the workload before you can trust autonomous behavior.”
—Scott McClure

Kaufmann added one key takeaway for watching this space of agentic AI and AppSec: “This needs to be treated as a living document, not a checklist with an expiration date. The best security teams will use it as a baseline and extend it as they learn.”

Learn how to secure your entire AI supply chain with an ML-BOM. RL’s Dhaval Shah explains how. 

*** This is a Security Bloggers Network syndicated blog from Blog (Main) authored by John P. Mello Jr.. Read the original post at: https://www.reversinglabs.com/blog/red-teaming-agentic-ai-key-takeaways-from-csas-new-guide