Process Monitoring: Detect Malicious Behavior in Your Containers

The default pod provisioning mechanism in Kubernetes has a substantial attack surface, making it susceptible to malevolent exploits and container breakouts. To achieve effective runtime security, your containerized workloads in Kubernetes require multi-layer process monitoring within the container. 

In this article, I will introduce you to process monitoring and guide you through a Kubernetes-native approach that will help you enforce runtime security controls and detect unauthorized access to host resources.

What is Process Monitoring?

When you run a containerized workload in Kubernetes, several layers should be taken into account when you begin monitoring the process within a container. This includes container process logs and artifacts, Kubernetes and cloud infrastructure artifacts, file system access, network connections, system calls required and kernel permissions (specialized workloads). Your security posture depends on how effectively your solutions can correlate disparate log sources and metadata from these various layers. Without effective workload runtime security in place, your Kubernetes workloads, which have a large attack surface, can easily be exploited by adversaries and face container breakouts.  

Traditional Monitoring Systems

Before I dive into the details on how to monitor your processes and detect malicious activities within your container platform, let us first take a look at some of the shortcomings of traditional monitoring systems in the context of Kubernetes. 

As previously mentioned, your security posture relies on your solution’s capability of weaving various log contexts together. Traditional monitoring solutions, such as endpoint detection and response (EDR) and endpoint protection systems, do not do this well for container platforms. The following are four reasons why traditional monitoring solutions are no longer sufficient for Kubernetes clusters:

Sponsorships Available

Sponsorships Available

Sponsorships Available

1. Cannot see underlays such as IPIP or VXLAN and are blind to traffic between containers.
2. Unaware of the process privileges and file permissions of containers accessing the underlying host.
3. May cause privilege escalation: Because traditional solutions are blind to the Kubernetes container runtime interface (CRI) and its mechanics and security issues, it may allow containers to access resources on the host.
4. Lack of container and container networking awareness: Increases the chance of false negatives on attackers’ lateral movement, as traditional monitoring systems can only view activity from the host’s perspective.

Kubernetes-Native Monitoring

Kubernetes-native monitoring involves monitoring processes using the various logs already available in Kubernetes. An example of this is shown in the figure below. As you can see, each process layer that makes up a containerized application comes with new monitoring and logging requirements. However, these layers also represent a new attack surface, and they require a completely different approach than the network and application monitoring traditional IT security teams are familiar with. This means new systems, more storage, and additional compute resources are needed. As a result, monitoring can become difficult and expensive.

To reduce monitoring costs and build defenses within each layer, it is best to use a Kubernetes-native monitoring solution. Many enterprises and open-source solutions can help you build a defense at each layer by offering the ability to target groups of layers with various tools and mechanisms, such as tracepoints, eBPF, ptrace and kprobes. I advise you to analyze your threat model and pick the solution that works best.

Below are eight monitoring techniques that provide a foundation for comprehensive container  runtime security and threat detection:

1. Routinely inspect Kubernetes audit logs to understand what Kubernetes API requests clients are making and detect suspicious behavior. 
2. Enable infrastructure logging from your cloud provider to detect suspicious activity in your cloud provider’s infrastructure.
3. Block processes spawned by individual containers or Kubernetes orchestration that creates containers.
4. Be aware of file system changes (e.g., binaries, packages) via monitoring file system reads and writes and implementing additional isolation through mandatory access control (MAC) to prevent privilege escalations.
5. The ability to monitor each container process’ kernel system calls.
6. Filter, block and alert suspicious kernel system calls to prevent containers from accessing host resources.
7. Enforce network policy and monitor every network connection (or socket) introduced by individual container processes.
8. Isolate a container using network policy or a node running the container, suspend it to investigate suspicious activity, and collect forensics data in Kubernetes. For Docker-based containers, the pause command can be used to achieve this.

Note: Since pausing a container will stop normal operations, this ability should only be used in response to serious events such as security incidents.

These techniques focus on monitoring to help you detect malicious behavior. If you would like to learn how to prevent malicious behavior by setting up controls and implementing existing Linux defenses closer to the container, be on the lookout for my second article on this subject.

Summary

There is no better way to secure your Kubernetes workloads from adversaries than to leverage Kubernetes-native monitoring to detect malicious activity. You should review your threat model in depth and implement process monitoring controls tailored to it. The techniques I offered in this article will serve as a solid foundation for your current or future solutions. If you want to learn more and view further examples, I recommend reading chapter four of Kubernetes Security and Observability: A Holistic Approach to Securing Containers and Cloud-Native Applications, an ebook I co-authored.

To learn more about new cloud-native approaches for establishing security and observability for containers and Kubernetes, check out this O’Reilly eBook, authored by Tigera.

To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2022 – October 24-28, 2022