Is ‘Secure Immutable Infrastructure’ a Myth?
The trend to shift left for security, when done right, has generated some positive results. As software development teams seek to deploy software at faster speeds, security teams have concurrently been tasked with making sure that compliance is met and that code is secure in such a way that software development speeds are not impeded, thanks largely through automation. Ideally, once deployed, the software and infrastructure, whether in cloud or on-premises environments, remain secure, ideally on so-called “immutable infrastructure.” Without going into detail about its computing structure, immutable infrastructure, described more than seven years ago by Chad Fowler, means that once software is deployed, no security patches, updates or reconfiguration are required.
However, unfortunately, there is ample evidence that in many cases, once software is developed and deployed, a number of security holes can occur or remain. As major data breaches have revealed, container and microservices vulnerabilities in this new age of Kubernetes have often been the source. The conclusion is that containers and running servers in so-called immutable infrastructure environments have required updates, thus negating by definition their status as immutable infrastructure.
While speaking at Black Hat USA 2020, SaltStack Chief Technology Officer and founder Thomas S. Hatch said the notion of immutable infrastructure, as well as the shift left in general, reflect philosophies and idealized political ideologies in general. “They paint a very picturesque and rosy world, in which everything that you have as a problem is going to magically melt away,” Hatch said. “But as we’ve learned over the years, the implementation of these philosophies almost always complicates situations, brings up new problems and illustrates new challenges that we have to work through.”
The widescale adoption of Kubernetes and microservices has often served to compound the security problem. As organizations often rush to shift to cloud-native environments, applications running on microservices and Kubernetes typically can require more updates than stateful environments do, while security practices in these environments are often neglected. In other words, an overzealous shift left by SecOps teams has left security and vulnerabilities untended and exposed.
Security holes and vulnerabilities are thus certainly not limited to what organizations thought were immutable infrastructures. As mentioned above, the shift left in security has, for many organizations, resulted in lingering post-deployment and infrastructure-related security holes.
“So much time is spent shifting that infrastructure to the left, tipping that infrastructure utilization over to development, involving more DevSecOps and more implementation of security auditing standards,” Hatch said. “When we talk a little bit also about what happens when we shift to the right for infrastructure, what we end up running into has more to do with dealing with new channels [such as microservices and Kubernetes].”
Additionally, neglected container stacks that, while once considered a part of immutable infrastructure, often remain in place. “There are incredibly long-lived containers that are out in the wild; I can’t begin to tell you how often we run into infrastructures and containers that have remained unchanged for years and years and not updated,” Hatch said. “This is because microservices are created that many organizations do not think need to be updated. The result that we run into is that the entire stack of the microservice application that lays on top of these microservices just sits inside your Kubernetes deployments.”
Without undermining the importance of introducing security processes and practices at the very beginning of the development cycle, maintaining security controls for applications and infrastructure during the post-deployment stage hinges on proper monitoring and alerts. Automation is also key, as a less time-consuming way to remediate low-hanging fruit security fixes that security tools should manage. Proper security management also involves intense collaboration and teamwork; instead of speaking about a perceived need to shift left or even to shift right, all DevOps teams—including developers as well as security teams—should be consistently collaborating throughout the entire development, deployment and post-deployment cycle. The end result is “significantly better-managed infrastructures,” Hatch said.
“Exposure to the right information for your teams is critical, so they can make better decisions. We need to be able to make sure that certain [infrastructure-related] information about microservices and components can flow back through the development, operations and security teams,” Hatch said. “This again allows us all to come back together and communicate on the same page about the big picture, instead of constantly getting stuck in chasing the next philosophies. We don’t want to chase philosophies, but we want to deploy applications and keep our businesses moving.”
