Permission denied trying to use rootless Podman + docker-compose + Traefik with podman.sock

Question

TL:DR: Trying to use rootless Podman with docker-compose through podman socket, and use a Traefik container (talking to podman socket) to proxy traffic to other containers, related to https://stackoverflow.com/a/73774327/1469083

I get permission denied errors, which I can fix with privileged container, which I don't want to use.

Setup

I am running on RHEL 8

$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.6 (Ootpa)

Podman came preinstalled, I added docker-compose ("standalone") and podman-docker:

$ curl -SL https://github.com/docker/compose/releases/download/v2.10.2/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
$ chmod a+x /usr/local/bin/docker-compose
$ sudo yum install podman-docker

And activated rootless podman socket so that podman and docker-compose can talk to each other:

$ systemctl --user enable podman.socket
$ systemctl --user start podman.socket
$ systemctl --user status podman.socket
$ export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock
$ echo $DOCKER_HOST
unix:///run/user/1001/podman/podman.sock

I also switched network backend to netavark, DNS did not work without that change

$ podman info |grep -i networkbackend
  networkBackend: netavark

Problems

First I tried the compose stack from https://stackoverflow.com/a/73774327/1469083 with small modifications:

version: "3"
services:
  frontend:
    image: "docker.io/traefik:v2.8"
    ports:
      - "3000:80"
      - "127.0.0.1:3080:8080"
    command:
      - --api.insecure=true
      - --providers.docker
    volumes:
      - /run/user/$UID/podman/podman.sock:/var/run/docker.sock

  backend:
    labels:
      traefik.http.routers.backend.rule: Host(`localhost`)
    image: "tomcat:latest"
    scale: 3

My setup did not appreciate the $UID variable:

WARN[0000] The "UID" variable is not set. Defaulting to a blank string. 
...
Error response from daemon: make cli opts(): error making volume mountpoint for volume /run/user//podman/podman.sock: mkdir /run/user//podman: permission denied

I replaced the volume map with hard-coded UID=1001 (it is the UID of the user running rootless podman, I assumed I should use that one?). Socket looks like this:

ls -la /run/user/1001/podman/podman.sock 
srw-rw----. 1 myrootlessuser myrootlessuser 0 22. 9. 11:28 /run/user/1001/podman/podman.sock


volumes:
  - /run/user/1001/podman/podman.sock:/var/run/docker.sock

But now I get permission denied errors from Traefik trying to connect to /var/run/docker.sock unsuccessfully:

example-docker-compose-frontend-1  | time="2022-09-22T12:04:52Z" level=error msg="Provider connection error Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version\": dial unix /var/run/docker.sock: connect: permission denied, retrying in 5.656635591s" providerName=docker

If I change the Traefik-container to privileged: true, this fixes the problem. I don't get the errors anymore, and proxying works like it should.

But, I would prefer not to use privileged containers for security reasons, or at least understand why it is like this.

Questions

1. How can I make this work with non-privileged Traefik container?

2. How do I verify that rootless docker/podman socket is working correctly? I've seen commands like this for testing rootful podman socket, but haven't had success on rootless

   $ sudo curl -H "Content-Type: application/json" --unix-socket /var/run/docker.sock http://localhost/_ping

   curl: (7) Couldn't connect to server

3. Where can I find the documentation for setting up docker socket for rootless Podman? Did I do my setup correctly (systemctl --user enable podman.socket etc.)? I've only been able to find some blogs about this, and advice varies and is often for older Podman versions. For example:

• https://fedoramagazine.org/use-docker-compose-with-podman-to-orchestrate-containers-on-fedora/ tells me to enable podman.socket
• https://earthly.dev/blog/earthly-podman/ tells me to enable podman.service

Answer 1

Question 1 and 2

If you are using

export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock

you are using rootless (unprivileged) Podman (even if you specify privileged: true in the Compose file).

To use the leaked socket in the container, you need to run podman run with the command-line option --security-opt label=disable.

Example:

Start and enable the podman socket

$ systemctl --user enable --now podman.socket
Created symlink /home/testuser/.config/systemd/user/sockets.target.wants/podman.socket → /usr/lib/systemd/user/podman.socket.

Test the Docker API service. Result: failure. Curl prints Couldn't connect to server

$ podman run --rm \
  -v $XDG_RUNTIME_DIR/podman/podman.sock:/var/run/docker.sock \
  docker.io/library/fedora \
    /usr/bin/curl \
      -H "Content-Type: application/json" \
      --unix-socket /var/run/docker.sock \
        http://localhost/_ping
curl: (7) Couldn't connect to server

Test the Docker API service again, but now add --security-opt label=disable. Result: success

$ podman run --rm \
  --security-opt label=disable \
  -v $XDG_RUNTIME_DIR/podman/podman.sock:/var/run/docker.sock \
  docker.io/library/fedora \
    /usr/bin/curl \
      -Hs "Content-Type: application/json" \
      --unix-socket /var/run/docker.sock \
        http://localhost/_ping
OK$

The Docker API service responded with the text string OK.

(In the command above I also added the curl option -s so that curl prints less debug output)

I would guess that adding privileged: true to the Compose file has the same effect as providing the --privileged command-line option to podman run.

One of the effects of using --privileged is that it implies --security-opt label=disable.

Question 3

Summary:

It is enough to run

$ systemctl --user start podman.socket

to set up the UNIX socket for rootless Podman.

It's not necessary but if you in addition to that run

$ systemctl --user start podman.service

the podman.service will be started right away (even before the first client has connected to the UNIX socket).

Longer version:

If podman.socket is active, then the podman.service will be started when a client connects. (Podman supports socket activation)

The podman.service will also be started after a reboot if the podman.service has been enabled (systemctl --user enable podman.service) and lingering is enabled (loginctl enable-linger).

The podman.service will also be started when the user logs in if the podman.service has been enabled (systemctl --user enable podman.service).

The podman process running in the podman.service will automatically exit after some time of inactivity (by default 5 seconds).

On a Fedora 36 computer, the Restart directive is set to no (the default value):

$ grep Restart= /usr/lib/systemd/user/podman.service
$ 

This means that it really doesn't matter much whether

systemctl --user enable podman.service

has been run or not. The service podman.service will anyway be stopped in 5 seconds if no clients access it.