Dockerfile does not receive environment variables

Question

Core problem is that when a container is creating files for persistence they are effectively owned by root and will require me to enter sudo password to delete. I want all containers to run as my user or at least in a way that I can delete temporary files created by containers. Look at this minimal example:

# docker-compose.yml
version: "2.2"
services:
    app:
        build: .
        container_name: app
        environment:
            - UID=${UID}
            - GID=${GID}
            - USER=${USER}

# Dockerfile
FROM alpine

RUN apk update
RUN apk upgrade
RUN apk add shadow

RUN useradd -G root,wheel -u ${UID} -g ${GID} -s /bin/ash -d /home/${USER} ${USER}
USER ${USER}
CMD /bin/ash

# output
❯ docker-compose up -d --remove-orphans --build
[+] Building 0.4s (8/8) FINISHED                                                                                                                                                                                                                  
 => [internal] load build definition from Dockerfile                                                                                                                                                                                         0.0s
 => => transferring dockerfile: 212B                                                                                                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                            0.0s
 => => transferring context: 32B                                                                                                                                                                                                             0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                                             0.0s
 => [1/5] FROM docker.io/library/alpine                                                                                                                                                                                                      0.0s
 => CACHED [2/5] RUN apk update                                                                                                                                                                                                              0.0s
 => CACHED [3/5] RUN apk upgrade                                                                                                                                                                                                             0.0s
 => CACHED [4/5] RUN apk add shadow                                                                                                                                                                                                          0.0s
 => ERROR [5/5] RUN useradd -G root,wheel -u ${UID} -g ${GID} -s /bin/ash -d /home/${USER} ${USER}                                                                                                                                           0.3s
------
 > [5/5] RUN useradd -G root,wheel -u ${UID} -g ${GID} -s /bin/ash -d /home/${USER} ${USER}:
#0 0.325 useradd: invalid user ID '-g'
------
failed to solve: executor failed running [/bin/sh -c useradd -G root,wheel -u ${UID} -g ${GID} -s /bin/ash -d /home/${USER} ${USER}]: exit code: 3

The useradd command is failing because none of the env vars are set. That means the command that is being run is /bin/sh -c useradd -G root,wheel -u -g -s /bin/ash -d /home.

Related SO answers / what I've tried so far:

Add environments to docker-compose.yml file this is exactly what I've done here.

Add -e option when running Dockerfile.

I've also tried to add environment variables in front of docker-compose and docker build commands like so:

UID=$UID GID=$GID USER=$USER docker-compose up --build --remove-orphans -d

and

UID=1000 GID=1000 USER=myusername docker-compose up --build --remove-orphans -d

Just for good measure I've also tried to user version 3 inside docker-compose.yml

I've also tried to put envirenment variables inside a .env file

USER=myusername
GID=1000
UID=1000

So I'm looking for an explanation for why it's not receiving anything and a suggestion for solutions to try.

As suggested to try -u option:

❯ docker build -t abc -u "$(id -u):$(id -g)" -f Dockerfile .
unknown shorthand flag: 'u' in -u
See 'docker build --help'.
❯ docker build --help

Usage:  docker build [OPTIONS] PATH | URL | -

Build an image from a Dockerfile

Options:
      --add-host list           Add a custom host-to-IP mapping (host:ip)
      --build-arg list          Set build-time variables
      --cache-from strings      Images to consider as cache sources
      --cgroup-parent string    Optional parent cgroup for the container
      --compress                Compress the build context using gzip
      --cpu-period int          Limit the CPU CFS (Completely Fair Scheduler) period
      --cpu-quota int           Limit the CPU CFS (Completely Fair Scheduler) quota
  -c, --cpu-shares int          CPU shares (relative weight)
      --cpuset-cpus string      CPUs in which to allow execution (0-3, 0,1)
      --cpuset-mems string      MEMs in which to allow execution (0-3, 0,1)
      --disable-content-trust   Skip image verification (default true)
  -f, --file string             Name of the Dockerfile (Default is 'PATH/Dockerfile')
      --force-rm                Always remove intermediate containers
      --iidfile string          Write the image ID to the file
      --isolation string        Container isolation technology
      --label list              Set metadata for an image
  -m, --memory bytes            Memory limit
      --memory-swap bytes       Swap limit equal to memory plus swap: '-1' to enable unlimited swap
      --network string          Set the networking mode for the RUN instructions during build (default "default")
      --no-cache                Do not use cache when building the image
      --pull                    Always attempt to pull a newer version of the image
  -q, --quiet                   Suppress the build output and print image ID on success
      --rm                      Remove intermediate containers after a successful build (default true)
      --security-opt strings    Security options
      --shm-size bytes          Size of /dev/shm
  -t, --tag list                Name and optionally a tag in the 'name:tag' format
      --target string           Set the target build stage to build.
      --ulimit ulimit           Ulimit options (default [])

System information

❯ lsb_release -a
LSB Version:    n/a
Distributor ID: ManjaroLinux
Description:    Manjaro Linux
Release:        21.3.0
Codename:       Ruah
❯ docker --version
Docker version 20.10.16, build aa7e414fdc
❯ docker-compose --version
Docker Compose version 2.6.0

Answer 1

When you're building an image, only the contents of the build: block are available. The environment: block is not available inside the Dockerfile, nor are volumes: mounts or networks: (including the automatic networks: [default]).

In principle you can make your Dockerfile work by declaring those parameters as ARG in the Dockerfile, and passing them in build: { args: } in the Compose file.

However, "which user is running this container" isn't something you usually want to build into your image – imagine having to rebuild tools from source whenever someone has a different user name or user ID. You can use the Compose user: directive to cause the container to run as a different user ID that you select at run time. For purposes of sharing files, this often needs to be a numeric user ID (the output of id -u). You do not need to do any setup in the image for this. The user won't exist in the container's /etc/passwd file, but the only consequence of this is usually a cosmetic complaint in some interactive shell prompts.

version: "2.2"
services:
    app:
        build: .
        user: 1000
        # volumes:
        #     - ./app_data:/data

In the Dockerfile, I'd suggest setting up a single dedicated directory to hold your application's writeable data (if that's required at all). It's good practice to create a non-root user, but it doesn't need a specific uid. Leave most files owned by root and not writeable by other users; do not RUN chown or RUN chmod. The volumes: mount shown above will replace the container directory with the host directory, including its (numeric) ownership.

FROM alpine

# Create the non-root user (BusyBox adduser syntax)
RUN adduser -S -D -H nonroot

# ... do the normal things to install and build your application ...
# (still as root; do not chown the files)

# Create the data directory
RUN mkdir /data && chown nonroot /data

# Switch to the non-root user only to run the actual container
USER nonroot
CMD ["the_program"]

Answer 2

ARG is different from ENV. The ARG is used when you run the docker build command and the ENV is used when you actually run the container from the image using the docker run command.

Basic Information on Working with Env and Args

Let's say I have the following dockerfile

FROM alpine:3.12

ARG UID
ARG USER

ENV ENV_UID=${UID}
ENV ENV_USER=${USER}

RUN echo "Build arg UID=${UID}"
RUN echo "Build arg USER=${USER}"
RUN echo "Environment variable UID=${ENV_UID}"
RUN echo "Environment variable USER=${ENV_USER}"

and docker compose file

services:
  app:
    build:
      context: .
      args:
        UID: ${UID1}
        USER: ${USER1}

    environment:
      UID: ${UID}
      USER: ${USER}

Role of Env File during Build

While running the docker file, it will give you warning because docker compose has no idea from where to pick the values for these variables at the build time.

$ docker-compose up --build
WARN[0000] The "UID1" variable is not set. Defaulting to a blank string. 
WARN[0000] The "USER1" variable is not set. Defaulting to a blank string. 
WARN[0000] The "UID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "USER" variable is not set. Defaulting to a blank string. 

Now you can provide environment file for passing these arguments (.env) to it which supports the variable expansion

UID1=$UID
USER1=$USER

Why does $UID is still not working? That is because it only reads the variables from your environment file and UID is not in the env but USER is.

The UID or EUID variable is provided by your bash, to which docker and docker compose is unaware.

How to fix this?

So you know that the docker compose knows about the environment variables, you can create the one which does't conflict with shell variables.

The new docker-compose.yaml would be

services:
  app:
    build:
      context: .
      args:
        UID: ${UID1}
        USER: ${USER1}

    environment:
      UID: ${ENV_UID} # updated
      USER: ${USER}

and the .env file would also have similar update

UID1=${ENV_UID} # updated
USER1=${USER}

Conclusion

Docker compose looks for the environment variables and has no information for the shell variables. The $UID variable is provided by the shell, you can check that by searching for "your shell name default variables" on your favourite search engine. To fix this, you are supposed to export an environment variable different from the shell variable, this is only to avoid conflicts.