Locking Down Kubernetes: RBAC, Audit Logs, etcd Security and More

 

 

What is Kubernetes Security?

Kubernetes is a highly extensible and portable container orchestration platform commonly used by enterprises and various industries worldwide. This rapidly expanding ecosystem helps manage container workloads and services at a massive scale. However, while some built-in features can help secure your Kubernetes workloads, complete coverage typically requires additional Kubernetes security tools and services. This makes Kubernetes security a critical part of modern information security strategies.

Kubernetes security risks correspond to the container lifecycle, requiring implementing measures against security risks in Kubernetes container environments. It involves applying security to threats at runtime, remediating known vulnerabilities during the build phase, and avoiding misconfigurations during the build and deploy phases. Employ these essential Kubernetes security best practices to secure cloud native applications and infrastructure.

How Can You Secure Your Kubernetes (K8s) Deployment?

Enable Kubernetes Role-Based Access Control (RBAC)

RBAC lets you define specific access to the Kubernetes API, and the permissions users have. Kubernetes 1.6 and later versions enable RBAC by default. However, when you manually enable RBAC, you need to disable the legacy attribute-based access control (ABAC). 

You should also prefer namespace-specific permissions rather than cluster-wide permissions, even for debugging purposes. You should allow access on a need basis, depending on the situation.

Secure Kubernetes API Server Authentication

The Kubernetes API is the primary access point for Kubernetes clusters. Service accounts and admins can access APIs using kubectl (the command-line utility), REST API calls, and various client SDKs. The Kubernetes API server hosts various APIs and serves as the core component of a Kubernetes cluster by granting access and ensuring the cluster is up and running.

Ideally, all API calls in a cluster should use the Transport Layer Security (TLS) protocol because it encrypts communication. You should also implement an API authentication mechanism for the API server, such as simple certificates and bearer tokens. Enterprise-level clusters typically segregate users into specific groups and control access by integrating third-party OpenID Connect (OIDC) providers or lightweight directory access protocol (LDAP) servers. 

Enable Audit Logging

Kubernetes can save granular records of activities performed in a cluster. These audit logs can help detect potential security issues in near real-time. You should enable audit logs for your Kubernetes clusters and monitor for suspicious activity and API calls. It can help you detect various malicious activities, such as brute force attacks to crack passwords.

Kubernetes disables audit logs by default. You can enable logs by using the Kubernetes audit policy to define the following audit levels:

  • None—asks Kubernetes not to log events matching this rule.
  • Metadata—requests to log certain metadata like the requesting user, resource, and timestamp.
  • Request—this option logs event metadata and request body. It does not log the response body and does not apply to non-resource requests.
  • RequestResponse—this audit level logs requests, response bodies, and event metadata. It does not apply to non-resource requests.

etcd Security

In Kubernetes, etcd is responsible for storing the cluster’s state and secrets. As a result, this sensitive resource is an attractive target for malicious actors. Unauthorized users with access to etcd can take over the Kubernetes cluster or use read access to elevate their privileges.

You can secure etcd by configuring TLS for etcd to use during client-server communication. Here are recommended configuration options:

 

cert-file= A certificate for SSL/TLS connections with etcd.
–key-file= A non-encrypted certificate key.
–client-cert-auth Tells etcd to check all incoming HTTPS requests to locate a client certificate signed only by a trusted certificate authority (CA).
–trusted-ca-file=<path> A trusted CA.
–auto-tls Specifies to use a self-signed auto-generated certificate for client connections.

 

Here are options for configuring TLS for etcd to use during server-to-server communication:

 

–peer-cert-file=<path> A certificate for SSL/TLS connections between peers.
–peer-key-file=<path> A non-encrypted certificate key.
–peer-client-cert-auth Tells etcd to check for valid signed client certificates on incoming peer requests.
–peer-trusted-ca-file=<path> A trusted CA.
–peer-auto-tls Specifies to use self-signed auto-generated certificates for peer-to-peer connections.

 

Private Kubernetes API Endpoint

You can configure the Kubernetes API endpoint of your cluster as part of a public or private subnet. When configuring a private cluster, ensure the API server in the control plane has a private IP address that denies access to the master from the public Internet. You can control the network access to the cluster API endpoint using security access control lists (ACLs) or network security settings to establish granularity.

Harden Node Security

You can harden node security using the following best practices:

  • Configuration benchmarks and standards—you should configure the host according to security recommendations and validate clusters against the benchmarks relevant to each Kubernetes release.
  • Admin access minimization—reducing the administrative access on your Kubernetes nodes can help minimize the attack surface area.
  • Node isolation and restrictions—you can run certain pods on relevant nodes or groups of nodes to ensure these pods run on nodes with proper isolation and security configurations.

You can control which nodes a certain pod can access by adding labels to node objects. It enables these pods to target certain nodes. You can use this option:

kubectl label nodes <node name> <label key>=<label value>

After applying the node label, you should add a nodeSelector to these pod deployments to ensure the pod deploys to the chosen node. Here is an example YAML file:

apiVersion: v1

kind: Pod

metadata:

  name: nginx

  labels:

    env: staging

spec:

  containers:

  - name: nginx-staging

    image: nginx

Conclusion

In this article, I explained the basics of Kubernetes security and how to lock down Kubernetes clusters:

  • Enable Kubernetes role-based access control (RBAC) – you should allow access on a need basis, depending on the situation.
  • Secure Kubernetes API server authentication – you should implement an API authentication mechanism for the API server, such as simple certificates and bearer tokens.
  • Enable audit logging – audit logs can help detect security issues in near real-time. You should enable audit logs to monitor for suspicious activity and API calls.
  • etcd security – you can secure etcd by configuring TLS for etcd to use during client-server communication.
  • Private Kubernetes API endpoint – ensure the API server in the control plane has a private IP address that denies access to the master from the public Internet.
  • Harden node security – implement configuration benchmarks and standards, admin access minimization, and perform node isolation and restrictions.

I hope this will be useful as you improve your Kubernetes security.

Avatar photo

Gilad David Maayan

Gilad David Maayan is a technology writer who has worked with over 150 technology companies including SAP, Oracle, Zend, CheckPoint and Ixia, producing technical and thought leadership content that elucidates technical solutions for developers and IT leadership.

gilad-david-maayan has 44 posts and counting.See all posts by gilad-david-maayan