Docker folder permissions work different on x86 pc compared to nvidia jetson

Question

We are facing a problem where folder permissions are not persistent across layers when using docker on nvidia jetson boards.

FROM ubuntu

RUN useradd -u 1000 vtc 
RUN echo "vtc:vtc" | chpasswd 
RUN mkdir -p /whatever/folder/
RUN chown -R vtc:vtc /whatever/folder/ && ls -la /whatever/folder/
RUN ls -la /whatever/folder/


WORKDIR /whatever/folder/


ENTRYPOINT [ "bash" ]

This is the minimal Dockerfile to describe our problem. When building this, we get this output on the nvidia board:

Step 1/8 : FROM ubuntu
 ---> 37f74891464b
Step 2/8 : RUN useradd -u 1000 vtc
 ---> Running in ff4af715f7e0
Removing intermediate container ff4af715f7e0
 ---> 50286a67966b
Step 3/8 : RUN echo "vtc:vtc" | chpasswd
 ---> Running in 7f8b7c7e4498
Removing intermediate container 7f8b7c7e4498
 ---> 0f3397d045b9
Step 4/8 : RUN mkdir -p /whatever/folder/
 ---> Running in 2376806263cb
Removing intermediate container 2376806263cb
 ---> ec761f93f3cf


Step 5/8 : RUN chown -R vtc:vtc /whatever/folder/ && ls -la /whatever/folder/
 ---> Running in b3b02071172c
total 12
drwxr-xr-x 1 vtc  vtc  4096 Aug  9 13:09 .
drwxr-xr-x 1 root root 4096 Aug  9 13:09 ..
Removing intermediate container b3b02071172c
 ---> 3c087553d180
Step 6/8 : RUN ls -la /whatever/folder/
 ---> Running in e2c7d95cfdbd
total 8
drwxr-xr-x 1 root root 4096 Aug  9 13:09 .
drwxr-xr-x 1 root root 4096 Aug  9 13:09 ..
Removing intermediate container e2c7d95cfdbd


 ---> bcf731209607
Step 7/8 : WORKDIR /whatever/folder/
 ---> Running in 911a365b139e
Removing intermediate container 911a365b139e
 ---> 887b446dee4e
Step 8/8 : ENTRYPOINT [ "bash" ]
 ---> Running in 8844d08364d0

As visible in 5/8, the permission on the folder are set to the correct user and group. However in 6/8 we print the permissions again and now they have changed to root:root . Since this is unexpected behavior, we repeated this on my laptop (x86 arch) with these results.

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 288B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/ubuntu:latest
#3 DONE 0.5s

#4 [1/7] FROM docker.io/library/ubuntu@sha256:0bced47fffa3361afa981854fcabcd4577cd43cebbb808cea2b1f33a3dd7f508
#4 CACHED

#5 [2/7] RUN useradd -u 1000 vtc
#5 DONE 0.2s

#6 [3/7] RUN echo "vtc:vtc" | chpasswd
#6 DONE 0.5s

#7 [4/7] RUN mkdir -p /whatever/folder/
#7 DONE 0.5s

#8 [5/7] RUN chown -R vtc:vtc /whatever/folder/ && ls -la /whatever/folder/
#8 0.472 total 12
#8 0.472 drwxr-xr-x 1 vtc  vtc  4096 Aug  9 12:48 .
#8 0.472 drwxr-xr-x 1 root root 4096 Aug  9 12:48 ..
#8 DONE 0.5s

#9 [6/7] RUN ls -la /whatever/folder/
#9 0.333 total 12
#9 0.333 drwxr-xr-x 1 vtc  vtc  4096 Aug  9 12:48 .
#9 0.333 drwxr-xr-x 1 root root 4096 Aug  9 12:48 ..
#9 DONE 0.4s

#10 [7/7] WORKDIR /whatever/folder/
#10 DONE 0.0s

Here we see that the permissions don't change from layer to layer.

On most other issues this because they use a COPY command or a volume, however with us this is not the case. We've cleaned all images and volumes, rebuild with no cache and still got the same permission problems.

Version on nvida jetson (Orin NX), jetpack 5.1.1.

Client:
 Version:           20.10.25
 API version:       1.41
 Go version:        go1.18.1
 Git commit:        20.10.25-0ubuntu1~20.04.1
 Built:             Fri Jul 14 22:01:07 2023
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.25
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.1
  Git commit:       20.10.25-0ubuntu1~20.04.1
  Built:            Thu Jun 29 21:55:06 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:        
 nvidia:
  Version:          1.1.7-0ubuntu1~20.04.1
  GitCommit:        629a689
 docker-init:
  Version:          0.19.0
  GitCommit:        

Version on x86, running on ubuntu 22.04

Client: Docker Engine - Community
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:51:00 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:51:00 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Answer 1

For testing, you can try merging the RUN commands to avoid any inter-layer inconsistencies.

FROM ubuntu

RUN useradd -u 1000 vtc && \
    echo "vtc:vtc" | chpasswd && \
    mkdir -p /whatever/folder/ && \
    chown -R vtc:vtc /whatever/folder/ && \
    ls -la /whatever/folder/

WORKDIR /whatever/folder/
ENTRYPOINT [ "bash" ]

That would make sure the commands are executed in a single layer, which can sometimes prevent unexpected behaviors related to layer caching.

Also, check if the storage driver remains the same in your different environments.

docker info | grep "Storage Driver"

If the storage drivers are different, you might consider changing the NVIDIA Jetson's Docker storage driver to overlay2 (if it is not already) to see if it resolves the inconsistency.