The Biggest Risks of AI Apps, Agents and Copilots – and How to Combat Them
Agentic AI offers untold benefits for productivity and efficiency, and unlike past technological trends, large enterprises are leading from the front. Employees today can use low-code development platforms to build their own AI agents and copilots using drag-and-drop interfaces and text-based natural language prompts. Less technical users – citizen developers – can develop them without the guardrails afforded by the software development lifecycle and they can create them quickly.
That carries risk, especially now with the general availability of Microsoft’s Copilot Studio and Salesforce’s Agentforce: People can now build their own AI agents that act autonomously on their behalf AND have access to massive amounts of data. Amazon Web Services explains it this way: “Humans set goals, but an AI agent independently chooses the best actions it needs to perform to achieve those goals.” So, even if you don’t read your emails or Teams messages – or they end up in the spam folder – Copilot reads them and can then use them to interact with you and anyone else across the enterprise.
This creates a Catch-22: Less technical people are building these AI agents, which means they’re prone to being built poorly or misconfigured, but they’re also quite powerful. Agentic AI is complex, and there’s no way of knowing which components of each agent are secure and which aren’t. Computers aren’t held accountable for their actions; humans are. That’s why organizations need to set guardrails for how these apps and agents are built.
The Serious Threat
Though these tools enhance productivity, many enterprises are still figuring out how to secure them properly. Today, someone can hack a copilot and do as they please within your organization. For instance, an attacker can intercept the file a copilot is reading to answer a question and change the answer. So, when you ask a copilot for a customer’s bank account information to trigger an agent to perform a wire transfer, an attacker can intercept that and send you back his or her account details.
Organizations using these tools face the risks of:
● End users giving AI apps/agents sensitive data. This could be solved by companies like Microsoft offering an opt-out checkbox that says, “We will not use your data to train our models.”
● End users maliciously giving AI apps/agents prompts to jailbreak them.
● Copilots accessing and returning files to end users that they shouldn’t.
● Copilots being overtaken by bad actors to perform social engineering/phishing attacks on end users.
Essentially, when someone hacks AI, they gain access to everything. Remote code execution (RCE) is a well-known vulnerability, but copilots and agents create a new vulnerability class: remote copilot execution, in which the attacker can take control of someone’s AI system and get it to operate on their behalf.
Someone can even socially engineer end users with remote control over an AI agent or copilot. If an employee is looking for, say, the company travel policy, the AI will send them an answer that links to a file, with references to it to build trust. However, if a bad actor is pulling the strings, they can have the agent direct the end user to a bad link that downloads malware onto their endpoint, takes control of their machine or downloads ransomware. All that the bad actor needs to do is use hidden instructions within an email, calendar invite or otherwise, that get picked up by the copilot. This is what’s referred to as promptware.
Post-compromise attackers living off the land of Microsoft Copilot can gain new capabilities they did not have before, including the ability to:
● Harvest credentials and collect sensitive data, abusing Copilot’s Retrieval-Augmented Generation (RAG) system to bypass DLP, identity protection and UEBA.
● Automate lateral movement by getting Copilot to spear phish all victim collaborators armed with the knowledge of previous interactions.
It’s not just enterprise-grade copilots, either. Recent research found that agents built on “Microsoft Copilot Studio can be misconfigured to expose sensitive corporate data and identities to the internet with no authentication.” Researchers found over 1,000 such bots and agents belonging to Fortune 500 companies, proving how easily these powerful AI systems can be misconfigured, making them ripe for attackers to manipulate and use at their will.
Organizations can’t afford to block AI-based tools these days; they must use this technology to maintain or create a competitive edge. Companies that get it right will reap the benefits. When you unleash these powerful and important tools, you also own the risk that comes with them.
Setting up Guardrails
There are infinite ways to string together words and prompts to trick an AI system. However, security teams can’t enumerate against human creativity, so trying to defend against prompt injections is essentially impossible. Instead, security teams must approach this problem from an application security perspective. You need to secure agent flows in real time to ensure that the AI is doing what its instructions tell it to do and not what hidden instructions or bad prompts tell it to do.
Increased visibility is critical. This includes not just how end users interact with agents and copilots, but also all the plugins and extensions used to integrate agents into third-party data sources, copilots, apps and systems. This should include visibility into how they are used and what real-time risks exist.
You’ll also need to conduct continuous risk assessments to determine which AI use cases and plugins have excessive permissions and/or access to sensitive data and whether any expose company secrets.
Use automated playbooks and mitigation actions so that when employees and third-party guest users customize and integrate plugins, they are built and integrated securely. Find solutions that can spot and prevent suspicious and malicious activities that use AI apps and agents as an attack surface – including data leakage – in runtime. You must also ensure promptware can’t get into your organization and be read by copilots. Regarding remediation, you’ll need an automated process that acts on detected threats and mitigates vulnerabilities with custom playbooks and granular policy authorization to ensure continuous and secure use of AI apps, agents and copilots.
Steering Copilots to Safety
The unchecked proliferation of AI apps, agents and copilots is a massive security red flag. They have become ubiquitous, and attackers have wasted no time finding the vulnerabilities to sneak into your network and interfere with these tools for their evil purposes. You can’t put this genie back in its bottle, so you must put the proper guardrails in place. Remember, there is no free lunch with AI. The upsides are tremendous, but security cannot be an afterthought.
