The AI Governance Gap Is Bigger Than We Think
One of the most dangerous phrases in technology is “we’ve got this under control.” It sounds reassuring. It signals maturity. It suggests that whatever challenge lies ahead has already been understood, measured and managed. Yet time and again, whether it was cloud computing, open source software, containers or Kubernetes, the industry has discovered that confidence and control are not the same thing. In fact, some of the largest security and operational failures of the last two decades occurred precisely because organizations believed they understood risks that were actually evolving much faster than their governance models.
That is why one statistic from JFrog’s newly released 2026 Software Supply Chain Security State of the Union report stood out more than any of the headline-grabbing numbers. According to the report, 97% of organizations claim to have AI governance in place. On its face, that sounds like good news. Given the speed at which AI has moved from experimentation into production systems, one would hope that enterprises are approaching governance seriously. But the deeper you dig into the report, the more that confidence begins to look questionable. The same research found that 53% of organizations are sourcing models from repositories where malicious payloads have already been detected. Even more concerning, 18% report having no governance over the IDEs and Model Context Protocol (MCP) servers embedded directly within developer workflows. Those numbers don’t suggest a mature governance environment. They suggest a growing disconnect between what organizations believe they control and what they actually control.
The headlines around the report will understandably focus on the dramatic rise in software supply chain attacks. Malicious npm packages increased by 451% over the past year, and JFrog identified more than 177,000 malicious packages across software registries. Those numbers are significant and should concern anyone responsible for software security. Yet focusing exclusively on malicious packages risks missing the larger story unfolding underneath the data. The software supply chain itself is changing. For years, supply chain security was primarily about understanding the provenance of source code, open source dependencies, containers and binaries. Organizations spent enormous amounts of time and money building visibility into those assets because they represented the primary pathways through which software entered production environments.
Today, however, software supply chains include far more than code. Modern development environments increasingly rely on foundation models, fine-tuned models, AI-generated code, agentic workflows, MCP servers, AI-powered IDE extensions and autonomous agents that can interact with enterprise systems on behalf of developers. Many organizations have adopted these capabilities with remarkable speed because the productivity benefits are real. Developers can write code faster. Teams can automate routine tasks. Agents can accelerate testing, troubleshooting and deployment activities. The problem is that governance frameworks have not expanded at the same pace as the technology itself. In many cases, organizations are applying governance models designed for software packages to ecosystems that now include autonomous systems making decisions and taking actions across development environments.
What makes the JFrog report particularly interesting is its attempt to quantify some of these emerging risks. The company identified 495 malicious AI models, 969 malicious AI agent skills and dozens of malicious OpenVSX extensions. Five years ago, these categories would barely have appeared on a security team’s radar. Today they represent legitimate attack vectors that can introduce risk long before software ever reaches a production environment. That shift matters because it reflects a broader evolution in attacker behavior. Historically, attackers focused on exploiting vulnerabilities within deployed applications. Over time, they moved upstream into software dependencies and package repositories. More recently, we have seen attacks targeting CI/CD pipelines and developer environments. AI is accelerating that trend by creating entirely new trust relationships that attackers can exploit.
Notice what all of these attack paths have in common. They are less about attacking software directly and more about attacking the mechanisms through which trust is established. A malicious package succeeds because someone trusts it enough to install it. A poisoned model succeeds because someone believes it is legitimate. A compromised MCP server succeeds because it is granted access to systems and workflows without sufficient scrutiny. Increasingly, the battle is not over code. It is over trust. That may sound like a subtle distinction, but it has enormous implications for how organizations approach security. Most governance frameworks were built around managing software artifacts. They were not designed to evaluate autonomous agents, model behavior or machine-generated actions occurring inside developer workflows.
This is also why another finding from the report deserves attention. More than 48,000 new CVEs were disclosed during 2025, a roughly 20% increase over the previous year. At the same time, JFrog’s researchers concluded that approximately two-thirds of the analyzed vulnerabilities had limited real-world applicability. For years, the security industry has conditioned itself to think in terms of vulnerability counts. More vulnerabilities meant more risk. More scanning meant better security. Larger dashboards suggested greater visibility. Yet many practitioners have quietly understood for years that vulnerability volume is often a poor proxy for actual risk. An exploitable trust relationship inside a developer workflow may be far more dangerous than dozens of low-impact CVEs sitting in a backlog waiting to be patched. The industry’s fixation on counting vulnerabilities may increasingly distract from understanding how attackers are actually operating.
That reality becomes even more apparent when examining the report’s findings around AI-generated code. Nearly half of the respondents indicated that reviewing and hardening AI-generated code has become a significant drain on security and development resources. This finding challenges one of the more simplistic narratives surrounding AI adoption. While AI can absolutely accelerate software development, it does not eliminate accountability. Every generated function still requires review. Every suggested dependency still needs validation. Every automated recommendation still needs oversight. In many organizations, AI is not reducing work so much as moving work from one part of the process to another. Developers may spend less time writing code, but security and platform teams often spend more time validating the output.
This is one reason I believe platform engineering is becoming increasingly important to the future of AI governance. Over the last several years, platform teams have evolved from managing Kubernetes environments to building internal developer platforms that standardize software delivery across organizations. Increasingly, they are also becoming responsible for governing how AI enters development workflows. Decisions about approved models, trusted MCP connections, agent permissions, policy enforcement and software provenance are increasingly landing on platform teams. In many organizations, platform engineering is quietly becoming the operational control plane for AI governance, whether executives recognize it or not.
The phrase JFrog uses throughout the report is “the illusion of mastery,” and I think that may ultimately prove to be the report’s most important contribution. The illusion is not that organizations lack governance entirely. Many have governance programs, security teams and compliance frameworks. The illusion is believing those mechanisms automatically provide visibility into a rapidly expanding ecosystem of models, agents, skills and autonomous systems. Governance only matters where work actually occurs. A policy document does not govern an AI model. A steering committee does not secure an MCP server. Real governance exists inside platforms, workflows and pipelines where decisions are made and actions occur.
Shimmy’s Take
What makes this report valuable is not the scary numbers. Security professionals see scary numbers every day. What makes it valuable is that it highlights a broader shift taking place across the industry. The software supply chain is no longer just about software. It is increasingly about trust. Trust in models. Trust in agents. Trust in extensions. Trust in autonomous systems operating inside developer environments. The organizations that succeed over the next several years will not necessarily be the ones with the most governance policies. They will be the ones that can demonstrate governance where software, AI and human decision-making actually intersect.
Right now, there is growing evidence that far fewer organizations have reached that point than they believe.
