Operation not permitted when performing a traceroute from a container deployed in Kubernetes [Linux capabilities]

Question

Trying to understand security context and capabilities in Kubernetes, I created following pod description:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  securityContext:
    runAsUser: 1000
  containers:
  - name: busy
    image: busybox
    command:
     - sleep
     - "3600"
    securityContext:
      runAsUser: 2000
      capabilities : 
        add: ["NET_ADMIN", "SYS_TIME"]

Where I am adding NET_ADMIN and SYS_TIME capabilities.

Given Linux capabilities man page: http://man7.org/linux/man-pages/man7/capabilities.7.html

I would expect to be able to perform a traceroute or set the date:

$ kubectl exec -it app -- traceroute google.fr
traceroute: socket: Operation not permitted
command terminated with exit code 1

$ kubectl exec -it app -- /bin/sh date --set="10:00:00"
date: can't set date: Operation not permitted

Since correct capabilities are set, I found strange that those operations are not permitted. Is this actually expected?

What is user `2000`? If you connect to the pod and run `whoami`, the user is not recognized: ```bash / $ whoami whoami: unknown uid 2000 ``` `traceroute` will work if you remove the `runAsUser: 2000` line from your pod specification. — George Tseres, Apr 05 '20 at 14:29
My objective was to run a traceroute without being root. This why I set the user but giving him the capabilities NET_ADMIN. I confirm removing runAsUser unable to make a traceroute work. — scoulomb, Apr 05 '20 at 17:14
So you specified in your container `runAsUser: 2000` but `busybox` as default don't have any user with this ID. As George Tseres mentioned, this user is not recognized because there is no user with this ID. Did you crate your own image and add there user? Did you follow any tutorials? Does it need to be busybox image? Are you using Kubeadm, Minikube, On-Prem? — PjoterS, Apr 06 '20 at 17:30
I was using busy box image as it is. Following your suggestion I tried with this image `echo 'FROM busybox RUN adduser --disabled-password --gecos "" MYUSER -u 2000 '> customBusybox.Dockerfile`. Where I define a user with UID 2000. Unfortunately when running my traceroute command I still have operation not permitted: ` $ k exec -it app -- /bin/sh / $ cat /etc/passwd | grep 2000 MYUSER:x:2000:2000::/home/MYUSER:/bin/sh / $ traceroute google.fr traceroute: socket: Operation not permitted`. What would you suggest? I setup a single kubernetes node on a VM and playing with capabilities. — scoulomb, Apr 08 '20 at 18:05

PjoterS · Answer 1 · 2020-04-23T20:15:19.963

In your example you are using Busybox.

Coming in somewhere between 1 and 5 Mb in on-disk size (depending on the variant), BusyBox is a very good ingredient to craft space-efficient distributions. BusyBox combines tiny versions of many common UNIX utilities into a single small executable. It provides replacements for most of the utilities you usually find in GNU fileutils, shellutils, etc. The utilities in BusyBox generally have fewer options than their full-featured GNU cousins; however, the options that are included provide the expected functionality and behave very much like their GNU counterparts. BusyBox provides a fairly complete environment for any small or embedded system.

I have tried to achieve what you want in many verious scenarios. Honestly, examples you choose to test securityContext here are not the best. I will post quite detailed information why.

To run traceroute or set date on busybox you need proper privileges. If you would use default busybox pod with root priviligies like below example It will works as expected.

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: busy
    image: busybox
    command:
     - sleep
     - "3600"

$ kubectl exec -ti app -- traceroute bbc.com
traceroute to bbc.com (151.101.128.81), 30 hops max, 46 byte packets
 1  10.32.1.1 (10.32.1.1)  0.006 ms  0.007 ms  0.003 ms
 2  216.239.48.36 (216.239.48.36)  5.476 ms  216.239.48.74 (216.239.48.74)  5.361 ms  216.239.48.36 (216.239.48.36)  4.669 ms
 ...
$ kubectl exec -ti app -- ping bbc.com
PING bbc.com (151.101.0.81): 56 data bytes
64 bytes from 151.101.0.81: seq=0 ttl=54 time=6.246 ms
64 bytes from 151.101.0.81: seq=1 ttl=54 time=6.081 ms

To run traceroute you need sudoprivileges`. For detailed information please check docs about traceroute on busybox.

As was mentioned in Kubernetes documentation regarding securityContext, in your YAML configuration you have set:

echo 'apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  securityContext:
    runAsUser: 1000 ## All containers in this pod will be run as user 1000
  containers:
  - name: busy
    image: busybox
    command:
     - sleep
     - "3600"
    securityContext:
      runAsUser: 2000 ## as you specified here, as default you will enter to this container as user with ID 2000
      capabilities : 
        add: ["NET_ADMIN", "SYS_TIME"]

In example above you have set runAsUser: 1000 which means that each container in this pod, default login will be as user 1000. Under container spec, you have set runAsUser: 2000 which means this specific container, as default will be logged in as user 2000.

To explain who is user 1000 please check this docs. In short this number is

Notice how the root user has the UID of 0. Most Linux distributions reserve the first 100 UIDs for system use. New users are assigned UIDs starting from 500 or 1000. For example, new users in Ubuntu start from 1000

Next thing I want to mention is Linux capabilities:

Output from BusyBox:

$ kubectl exec -ti app /bin/sh
/ # capsh --print
/bin/sh: capsh: not found

Output from Ubuntu:

$ kubectl exec -ti ubuntu /bin/bash
root@ubuntu:/# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
...
root@ubuntu:/# capsh --print
Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap

If you want to use capabilites, you will not achieve it using Busybox. If you want some extra information about traceroute for linxu, please check this link.

As last test for using traceroute I've created random user in container.

Ubuntu (default ubuntu image didn't have traceroute, needed to install it. apt-get update to update repository and then apt-get install traceroute:

$ kubectl exec -ti ubuntu /bin/sh
# whoami
root
# traceroute bbc.com
traceroute to bbc.com (151.101.0.81), 30 hops max, 60 byte packets
 1  10.32.1.1 (10.32.1.1)  0.032 ms  0.008 ms  0.007 ms
 2  209.85.253.197 (209.85.253.197)  6.294 ms 216.239.48.74 (216.239.48.74)  5.613 ms 216.239.48.36 (216.239.48.36)  5.335 ms
# useradd -m test    
# passwd test
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
# su test
$ whoami
test
$ id  
uid=1000(test) gid=1000(test) groups=1000(test)
$ traceroute bbc.com
traceroute to bbc.com (151.101.64.81), 30 hops max, 60 byte packets
 1  10.32.1.1 (10.32.1.1)  0.034 ms  0.008 ms  0.008 ms
 2  216.239.48.36 (216.239.48.36)  5.515 ms 216.239.51.111 (216.239.51.111)  5.494 ms 216.239.48.36 (216.239.48.36)  5.591 ms

Busybox:

$ kubectl exec -ti app /bin/sh
/ # whoami
root
/ # traceroute bbc.com
traceroute to bbc.com (151.101.192.81), 30 hops max, 46 byte packets
 1  10.32.1.1 (10.32.1.1)  0.005 ms  0.006 ms  0.003 ms
 2  216.239.48.36 (216.239.48.36)  5.453 ms  216.239.48.74 (216.239.48.74)  4.812 ms  209.85.252.4 (209.85.252.4)  6.787 ms
/ # adduser test
Changing password for test
New password:
Retype password:
passwd: password for test changed by root
/ # su test
/ $ whoami
test
/ $ id
uid=1000(test) gid=1000(test) groups=1000(test)
/ $ traceroute bbc.com
traceroute: socket: Operation not permitted

In short story, to execute traceroute in Busybox you need have root privileges. To run traceroute on ubuntu you have to pre-install traceroute command.

Regarding changing date in container, please check this tread.

This helps me understand a bit more. I agree that for busybox/alpine image it is not possible to run a traceroute when not root whatever the capabilities. However it is possible to drop capabilities and prevent root user from doing a traceroute. However I do not fully agree with last part of your answer because it seems that with the ubuntu image even when explicitly dropping ["NET_RAW", "NET_BIND_SERVICE", "NET_ADMIN"], with root or non root user I can still perform a traceroute. Made some proof here: https://github.com/scoulomb/myk8s/blob/master/Security/0-capabilities-bis-part1-basic.md — scoulomb, Apr 27 '20 at 20:17
Could you elaborate with what you are not agree? On Ubuntu you can perform traceroute as defined user (`test` user was created in example) which dont have root permissions. Ive mentioned that with default `ubuntu` image Ive used for this example didnt have `traceroute` command. Needed to install it. Another thing, in this link you are using alpine image, not busybox. — PjoterS, Apr 28 '20 at 11:23
Sorry if there is some misunderstanding on my side. When reading I have the feeling it is working on Ubuntu due to capabilities unlike busybox, but capabilities have actually no effect for Ubuntu image whatever is the user (unlike busybox where dropping `NET_RAW` capa for root prevents him from doing a traceroute). This part in particular confused me: `$ kubectl exec -ti app /bin/sh / # capsh --print /bin/sh: capsh: not found`, actually busybox has capabilities (we can see doing `grep Cap /proc/1/status`, and `capsh --decode=` ). Except that the answer is perfect :) . wdyt? — scoulomb, Apr 28 '20 at 17:17

Operation not permitted when performing a traceroute from a container deployed in Kubernetes [Linux capabilities]

1 Answers1