Using Podman with docker-compose, how to get multiple replicas of a service

Question

EDIT: weird DNS behavior was some kind of transient issue, and now RHEL/podman works the same way as Ubuntu/podman. I can't reproduce the issue, which makes most part (not 100% though) of this question moot.

I am trying to use Podman and docker-compose to create a compose stack with multiple replicas of backend container, and having a hard time with it.

I use Podman because I have to (comes from Red Hat platform), and picked docker-compose because it is familiar and I use in local dev host, too. I know there are alternatives (podman-compose etc). I learned that Podman 4.1 supported Docker Compose so this sounded like a good candidate.

As an example I have docker-compose.yml with one frontend container and 3 backend containers:

version: "3"
services:
  frontend:
    image: "nginx:latest"
    ports:
      - "3000:80"
    depends_on:
      - backend
  backend:
    image: "tomcat:latest"
    ports:
      - "8180-8280:8080"
    scale: 3

Note: this stack is just an example. It's purpose is only to highlight the networking aspects of multi-replica docker-compose . I could use something else than nginx:latest, and using e.g. Traefik can solve some of the problems...but sometimes you wish to connect directly from one container to a service with multiple container replicas.

Docker & docker-compose (Ubuntu)

Running this on host which has docker and docker-compose is straightforward.

Backend containers get assigned random ports from range 8180-8280.

$ docker ps
CONTAINER ID   IMAGE           COMMAND                  CREATED          STATUS          PORTS                                       NAMES
98941117708c   nginx:latest    "/docker-entrypoint.…"   25 seconds ago   Up 22 seconds   0.0.0.0:3000->80/tcp, :::3000->80/tcp       example-docker-compose-frontend-1
3d749e25eaba   tomcat:latest   "catalina.sh run"        26 seconds ago   Up 23 seconds   0.0.0.0:8193->8080/tcp, :::8193->8080/tcp   example-docker-compose-backend-1
854ba8f60cb3   tomcat:latest   "catalina.sh run"        26 seconds ago   Up 23 seconds   0.0.0.0:8192->8080/tcp, :::8192->8080/tcp   example-docker-compose-backend-2
e57e32181e8e   tomcat:latest   "catalina.sh run"        26 seconds ago   Up 23 seconds   0.0.0.0:8194->8080/tcp, :::8194->8080/tcp   example-docker-compose-backend-3

Logging into frontend container, service name backend resolves to all 3 backend containers

dig backend

; <<>> DiG 9.16.27-Debian <<>> backend
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31602
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;backend.           IN  A

;; ANSWER SECTION:
backend.        600 IN  A   172.19.0.3
backend.        600 IN  A   172.19.0.2
backend.        600 IN  A   172.19.0.4

curl backend:8080 works

Podman & docker-compose (RHEL 8)

Podman came preinstalled, I added docker-compose (standalone) and podman-docker:

# curl -SL https://github.com/docker/compose/releases/download/v2.10.2/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
# chmod a+x /usr/local/bin/docker-compose
# sudo yum install podman-docker

And activated rootless podman socket so that podman and docker-compose can talk to each other:

# systemctl --user enable podman.socket
# systemctl --user start podman.socket
# systemctl --user status podman.socket
# export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock

I also switched network backend to netavark, DNS did not work without that change

$ podman info |grep -i networkbackend
  networkBackend: netavark

1. Port ranges are not supported

Podman does not like ports: - "8180-8280:8080" due to this bug: https://github.com/containers/podman/issues/15111

[+] Running 1/0
 ⠿ Network example-docker-compose_default      Created                           0.0s
 ⠋ Container example-docker-compose-backend-3  Creating                          0.0s
 ⠋ Container example-docker-compose-backend-1  Creating                          0.0s
 ⠋ Container example-docker-compose-backend-2  Creating                          0.0s
Error response from daemon: make cli opts(): strconv.Atoi: parsing "8180-8280": invalid syntax

2. Without port range, address already in use conflict

Changed docker-compose.yml to remove port range

ports:
  - "8080:8080"

This results in port conflict, all 3 backend containers try to bind to 8080

Catalina.startup.Catalina.start Server startup in [190] milliseconds
Error response from daemon: rootlessport listen tcp 0.0.0.0:8080: bind: address already in use

3. Scale = 1

Let's try with just one backend container. System starts up

CONTAINER ID  IMAGE                            COMMAND               CREATED        STATUS            PORTS                   NAMES
217c3e726431  docker.io/library/tomcat:latest  catalina.sh run       7 seconds ago  Up 6 seconds ago  0.0.0.0:8080->8080/tcp  example-docker-compose-backend-1
9c3f86676bde  docker.io/library/nginx:latest   nginx -g daemon o...  6 seconds ago  Up 6 seconds ago  0.0.0.0:3000->80/tcp    example-docker-compose-frontend-1

DNS from frontend server looks weird. What are all these different backend IP addresses 10.89.0.3 - 10.89.0.12? Only the last of them works when I curl 10.89.0.x. Still, curl backend:8080 works fine?

4. Scale up from command line

I remove ports and scale from docker-compose.yml and start compose stack with scale=3 option:

version: "3"
services:
  frontend:
    image: "nginx:latest"
    ports:
      - "3000:80"
    depends_on:
      - backend
  backend:
    image: "tomcat:latest"

$ docker-compose up --scale backend=3
[+] Running 4/4
 ⠿ Container example-docker-compose-backend-2   Recreated                        0.3s
 ⠿ Container example-docker-compose-backend-3   Recreated                        0.2s
 ⠿ Container example-docker-compose-backend-1   Recreated                        0.3s
 ⠿ Container example-docker-compose-frontend-1  Recreated                        0.3s
Attaching to example-docker-compose-backend-1, example-docker-compose-backend-2, example-docker-compose-backend-3, example-docker-compose-frontend-1

Now compose stack starts nicely with 3 backend containers

$ docker ps
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
CONTAINER ID  IMAGE                            COMMAND               CREATED         STATUS             PORTS                 NAMES
b81c44246ae7  docker.io/library/tomcat:latest  catalina.sh run       22 minutes ago  Up 22 minutes ago                        example-docker-compose-backend-3
814dc5d307f7  docker.io/library/tomcat:latest  catalina.sh run       22 minutes ago  Up 22 minutes ago                        example-docker-compose-backend-2
fb0a5090a456  docker.io/library/tomcat:latest  catalina.sh run       22 minutes ago  Up 22 minutes ago                        example-docker-compose-backend-1
c0219d7fded4  docker.io/library/nginx:latest   nginx -g daemon o...  22 minutes ago  Up 22 minutes ago  0.0.0.0:3000->80/tcp  example-docker-compose-frontend-1

DNS from frontend has even more entries

# dig backend

; <<>> DiG 9.16.27-Debian <<>> backend
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29526
;; flags: qr rd ad; QUERY: 1, ANSWER: 11, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 410008265c2b0edb (echoed)
;; QUESTION SECTION:
;backend.           IN  A

;; ANSWER SECTION:
backend.        86400   IN  A   10.89.0.3
backend.        86400   IN  A   10.89.0.4
backend.        86400   IN  A   10.89.0.6
backend.        86400   IN  A   10.89.0.7
backend.        86400   IN  A   10.89.0.15
backend.        86400   IN  A   10.89.0.16
backend.        86400   IN  A   10.89.0.18
backend.        86400   IN  A   10.89.0.19
backend.        86400   IN  A   10.89.0.20
backend.        86400   IN  A   10.89.0.21
backend.        86400   IN  A   10.89.0.22

And curl backend:8080 does not work (not sure which port I should use now)

Questions

1. What's going on here?
2. Can I achieve a setup of 3 backend containers, so that DNS name backend would resolve to those, with podman & docker-compose?
3. Podman seems to support docker-compose (or vice versa), but only to a degree. Is there some documentation which tells what docker-compose features are supported on Podman, and which are not?

Answer 1

Podman is my container runtime of choice...unless I'm working with docker-compose, in which case I have found it to be "close but not quite" in terms of its docker API support.

However, for what you're trying to do, you could replace Nginx with Traefik, and let Traefik handle the load balancing. Traefik is a dynamic proxy that uses the Docker API and container labelling to discover containers and configure the proxy rules.

For example:

version: "3"
services:
  frontend:
    image: "docker.io/traefik:v2.8"
    ports:
      - "3000:80"
      - "127.0.0.1:3080:8080"
    command:
      - --api.insecure=true
      - --providers.docker
    volumes:
      - /run/user/$UID/podman/podman.sock:/var/run/docker.sock

  backend:
    labels:
      traefik.http.routers.backend.rule: Host(`localhost`)
    image: "quay.io/larsks/demoserver:latest"
    scale: 3

Here we're mapping all requests for Host: localhost to to our backend containers. This is just for the purposes of a demonstration (since I'll be running curl localhost/...); a more realistic configuration would use specific hostnames, or paths, etc. You can read more in the Routing configuration section of Traefik's Docker documentation, and also in the general Router documentation.

With this configuration, we see the following containers running:

$ podman ps
CONTAINER ID  IMAGE                             COMMAND               CREATED        STATUS            PORTS                                           NAMES
c29de1d137e2  docker.io/library/traefik:v2.8    --api.insecure=tr...  5 seconds ago  Up 6 seconds ago  0.0.0.0:3000->80/tcp, 127.0.0.1:3080->8080/tcp  demoserver_frontend_1
f4fac24cb494  quay.io/larsks/demoserver:latest  /usr/local/bin/st...  5 seconds ago  Up 6 seconds ago                                                  demoserver_backend_2
d9d388202be2  quay.io/larsks/demoserver:latest  /usr/local/bin/st...  5 seconds ago  Up 5 seconds ago                                                  demoserver_backend_1
5a3e6330739d  quay.io/larsks/demoserver:latest  /usr/local/bin/st...  5 seconds ago  Up 5 seconds ago                                                  demoserver_backend_3

And we can see that requests on port 3000 cycle between the available backends. Running this script:

for i in {1..10}; do
  curl http://localhost:3000/hostname
done

Produces as output:

5a3e6330739d
d9d388202be2
f4fac24cb494
5a3e6330739d
d9d388202be2
f4fac24cb494
5a3e6330739d
d9d388202be2
f4fac24cb494
5a3e6330739d