Docker network architecture: isolated container, host based routing

Question

As i play from time to time with docker, i hang on a problem where i can't get any further

I want to create a public "cloud" where a user can create its own container. The container is just a http app, nextcloud for example.

What would be the best way to isolate the container from each other and allow communication through the docker host to the internet, so if necessary i can block per firewall some IPs.

This should apply to both, incoming and outgoing connections.

I created a setup for this, described here: docker macvlan - no route to host (container) but could not even get the basic setup working properly.

For better understanding i created a diagram:

On the docker host a reverse proxy running which looks for a specific http header and forwards the requests based on that header to the wanted container.

The biggest problem is the network setup: i dont know how docker, the host and what else need to be configured to get this working.

Networking under linux is not my strength.

EDIT: Perhaps i ran into a XY problem (https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem)

Answer 1

I might be doing things a little too easy but I don't think you need to go with macvlan for this. A normal network per container is enough to separate all containers. All communication with the host will simply happen via gateway.

I just ran a couple of tests for the fun of it:

Setup:

rm -f addresses.txt
for i in $( seq 1 5 ) ; do 
    docker network create test$i
done

for i in $( seq 1 5 ) ; do 
    docker run -d --rm \
        --name myspecialtest$i \
        --network test$i \
        alpine sleep 3600

    docker exec -it myspecialtest$i ifconfig eth0 \
        | grep -P -o 'inet addr:\d+\.\d+.\d+.\d+' \
        | cut -d: -f2 >> addresses.txt
done

Test:

for i in $( seq 1 5 ) ; do 
    for ipaddress in $( cat addresses.txt ); do
        docker exec -it myspecialtest$i \
            ping -w 5 $ipaddress
    done
done

Clean up:

for i in $( seq 1 5 ) ; do
    docker container rm -f myspecialtest$i
    docker network rm test$i
done

As you can see the containers will be only able to ping themselves but not the other containers.