Container Security: Winter is Coming — Dress in Layers!
Recently I had the pleasure of joining hundreds of DevOps pros, IT managers, and security engineers at the first ever Container Security event at LEGOLAND. Attendees discussed the newest technologies, scariest threats, and biggest trends in the evolving world of container security. If you weren’t lucky enough to be a part of the fun, here’s a quick recap of what Threat Stack’s Director of Product, Todd Morneau, spoke about.
Container Security: Winter is coming; dress in layers!
Security is becoming an important feature of containerization. Not just for container companies like Docker, but for the Linux community, cloud platforms, and of course, container orchestrators such as Swarm and Kubernetes. Security features are obviously important, but that doesn’t mean infrastructure engineers take the time to actually use them to harden defenses at every layer of their stack.
Process Isolation and Access Control
Containers standardize deployment, allow developers to choose whichever library/framework best suits them, and scale out well with proper orchestration. Container technologies provide these benefits through the way they relate to the host system, by isolating processes while sharing a common kernel.
Different containers may have their own CPU, memory, and IO constraints, but they interface with shared host operating systems.
Docker adds a level of isolation that can contribute to a defense-in-depth strategy, but achieving robust security requires a thorough understanding of least-privilege access to both containers and hosts. User and group access controls must be audited, as well as Linux capabilities loaded at runtime. Visibility into user behavior at runtime is another challenge, but adhering to the principle of least-privileged access is a solid start.
Past Issues
In the past, security concerns were largely focused on kernel exploits and Docker’s integration with the kernel, since this is where most exploits happen. These concerns were well warranted, especially since many people run container processes as root.
Yet container adoption hasn’t halted or even slowed down. Thanks to developer ease-of-use, containers are becoming more widespread. Adopting containers requires you to think differently about security, although existing best practices still apply. Let’s take a look at some historical common vulnerabilities and exposures (CVEs) before discussing how to secure your stack.
Current Past Issues
There are two noteworthy, relevant types of CVEs: Linux kernel-specific issues that apply to any container runtime and container runtime-specific issues.
On the kernel side, it’s noteworthy that the top five Linux CVEs for 2018 are DoS attacks. If these sound like old issues, you’re right, but they’re still hanging around the kernel. If the host operating system is vulnerable, then your containers are already sunk.
A skim through the Docker-specific issues reveals that the majority target privilege escalation. Privileged containers can let attackers run arbitrary code, but lower layers may be exploited too. While containers can contribute to a defense-in-depth strategy, they can also add attack surface for exploiting kernel vulnerabilities.
Sometimes people think that containers are a cure-all for security threats. When set up correctly, they do provide strong security; however, they cannot guard against all possible attack vectors.
The easiest thing to do is to introduce security as part of the regular DevOps process, and container orchestration can help a great deal here. For example: deploying your security tool as a Kubernetes DaemonSet, which ensures that all nodes run a copy of your security-specific pod. This allows your security tool to be deployed automatically and by default when new worker nodes are established.
Stay Frosty
Attacks that achieve escape from the container are the most serious and require the most attention. Luckily, preventing escape is easy enough in the majority of cases, as long as you’re aware of CVEs and are applying patches/updates accordingly.
Bear in mind that the underlying kernel technologies are much older than modern container technologies. Container runtimes are changing much faster than the kernel, so new exploits may be found across a range of kernel versions. This makes it all the more important to do routine runtime checks of your containers and configuration checks of Docker (CIS, etc.).
Best Practices
Some recommendations apply to all containers — regardless of application or environment. First, mitigate against privileged access, both inside and outside the container. AWS has its own identity/access service (IAM) for hosts, and Kubernetes has its equivalent (RBAC). It’s worth the time to thoroughly understand these features because they form the new security perimeter for cloud infrastructure. At runtime, avoid Docker’s –privileged flag if at all possible. Processes in containers should not run as root. Dockerfiles should include a USER instruction, specifying what user to apply when the container is started.
The Docker security guide explains the connection: “This means that even if an intruder manages to escalate to root within a container, it will be much harder to do serious damage, or to escalate to the host.”
Other recommendations are more Kubernetes specific. First, pay special attention to configuring the Kubernetes API. A range of access controls and authorization checks that are critical to security apply in different contexts. Like the documentation says, “As Kubernetes is entirely API driven, controlling and limiting who can access the cluster and what actions they are allowed to perform is the first line of defense.”
Another important Kubernetes component is etcd. It serves as the backend for the API, and so access to etcd should be restricted as much as possible. The same documentation goes on to say, “Write access to the etcd backend for the API is equivalent to gaining root on the entire cluster, and read access can be used to escalate fairly quickly.”
Several places to focus initial Kubernetes security efforts.
Getting Started . . .
The practices outlined above will help you get started on securing your stack, but be warned — security is not a one-time concern. It requires constant vigilance to stay up to date with CVEs and patch software. Correlating information on vulnerabilities and misconfigurations with behavior into the environment at runtime is the next challenge.
To gain complete visibility into your infrastructure, it is important to obtain host runtime visibility as well as container runtime visibility. Integrating the native monitoring and logging tools at both levels is one approach for people with the time and resources to do it right. But many companies need to move fast and can’t wait for security to catch up. So they’ll choose to work with a vendor to tie it all together. At Threat Stack, we do this for customers every day.
And that’s it for Todd’s presentation. For more on Kubernetes security, see Todd’s recent article, “Why Kubernetes is Not a Silver Bullet.”
*** This is a Security Bloggers Network syndicated blog from Blog – Threat Stack authored by Hannah Klemme. Read the original post at: https://www.threatstack.com/blog/container-security-winter-is-coming-dress-in-layers
