323

I'm running Jenkins inside a Docker container. I wonder if it's ok for the Jenkins container to also be a Docker host? What I'm thinking about is to start a new docker container for each integration test build from inside Jenkins (to start databases, message brokers etc). The containers should thus be shutdown after the integration tests are completed. Is there a reason to avoid running docker containers from inside another docker container in this way?

David Maze
  • 130,717
  • 29
  • 175
  • 215
Johan
  • 37,479
  • 32
  • 149
  • 237
  • 19
    Another possibility is to mount the docker socket from the host as a volume in the container. That lets you create "sibling" containers and has the advantage of being able to reuse the cache. – Adrian Mouat Jan 10 '15 at 19:59
  • 8
    I've found that when using the docker socket from the host that in cases where I want to mount external volumes it's necessary to set the volume path relative to the host as that is where the docker daemon runs. Setting it relative to the container that starts containers will not necessarily work unless paths coincide. – Jakob Runge Mar 01 '16 at 15:50

5 Answers5

372

Running Docker inside Docker (a.k.a. dind), while possible, should be avoided, if at all possible. (Source provided below.) Instead, you want to set up a way for your main container to produce and communicate with sibling containers.

Jérôme Petazzoni — the author of the feature that made it possible for Docker to run inside a Docker container — actually wrote a blog post saying not to do it. The use case he describes matches the OP's exact use case of a CI Docker container that needs to run jobs inside other Docker containers.

Petazzoni lists two reasons why dind is troublesome:

  1. It does not cooperate well with Linux Security Modules (LSM).
  2. It creates a mismatch in file systems that creates problems for the containers created inside parent containers.

From that blog post, he describes the following alternative,

[The] simplest way is to just expose the Docker socket to your CI container, by bind-mounting it with the -v flag.

Simply put, when you start your CI container (Jenkins or other), instead of hacking something together with Docker-in-Docker, start it with:

docker run -v /var/run/docker.sock:/var/run/docker.sock ...

Now this container will have access to the Docker socket, and will therefore be able to start containers. Except that instead of starting "child" containers, it will start "sibling" containers.

gotgenes
  • 38,661
  • 28
  • 100
  • 128
predmijat
  • 3,844
  • 2
  • 11
  • 2
  • 1
    How to run docker commands without `sudo` when doing like this ? Thanks – c4k Oct 21 '16 at 17:18
  • 8
    You need to add user to `docker` group: `sudo usermod -aG docker $USER`. You'll need to relog after that. – predmijat Dec 04 '16 at 09:12
  • 3
    How to relog from within a cointainer? – thiagowfx Jul 10 '17 at 20:45
  • what about on MacOS machines? what is the analogue to `-v /var/run/docker.sock:/var/run/docker.sock` on MacOS? thx – Alexander Mills Aug 30 '17 at 23:55
  • 1
    @AlexanderMills It is the same because your docker socket is also located at `/var/run/docker.sock` when you run docker for mac on your macos machine. – Bruce Sun Aug 31 '17 at 05:01
  • 3
    what about windows? i dont have `/var/run/docker.sock` – Abdelhafid Nov 11 '17 at 09:05
  • 1
    Currently this not possible with Windows. docker.sock only exists on Linix/UNIX based systems. TCP is the only functional communication channel between containers on Windows. The tcp socker for the engine is tcp://localhost:2375. – wheredidthatnamecomefrom Nov 27 '17 at 17:14
  • 1
    Running 'docker ps' also lists , dockers running inside the docker ? – avee137 Jun 07 '18 at 15:38
  • 1
    even after sharing `-v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/:/var/lib/docker/ -v /usr/bin/docker:/usr/bin/docker ` I get `Cannot connect to the Docker daemon. Is the docker daemon running on this host?` It'd be great if anyone could help on this – Keerthana Prabhakaran Jun 21 '18 at 14:28
  • @KeerthanaPrabhakaran it sounds like you need to start the docker deamon on the host. – Ryan Fisher Jun 22 '18 at 20:31
  • 1
    You can also mount the docker binary from the host as all you need is the CLI (`-v /usr/local/bin/docker:/usr/local/bin/docker`). The article from Jérôme Petazzoni no longer recommends mounting the `docker` binary from the host, however I have been doing this on my Jenkins CI server and locally on MacOS without issue. I thought this change made the `docker` binary atomic? https://github.com/moby/moby/pull/20639. Otherwise you can download the distributable binary from here: https://download.docker.com/linux/static/stable/x86_64/ – Ryan Fisher Jun 22 '18 at 21:58
  • 1
    @predmijat It is a bad advice to suggest adding a user to the docker group. It's the same as giving your user root privileges. Do a search on 'docker group considered harmful'. – szczepanpp Jun 29 '18 at 23:14
  • Is it possible to do this on a windows docker host? – Darrell Jul 02 '18 at 13:25
  • You are absolute awesome!! You saved my life!! I've using `--privileged` and `-v /var/run/docker.socks:/var/run/docker.socks` for integrate docker in docker and start the service at the very moment I start the docker container – Marslo Aug 07 '18 at 03:47
  • However, [trying to create Docker bridge networks in the inner Docker does not work](https://stackoverflow.com/a/55653748/545127). – Raedwald Apr 12 '19 at 14:42
  • Is there a way to create sibling containers with something like `-v /var/run/docker.sock:/var/run/docker.sock` in k8s? – Dagang Nov 12 '19 at 00:23
  • When I use Jenkins' Docker support, it defaults to running all containerized jobs as a limited user (1001:1001), and these non-existent users can NOT write to /var/run/docker.sock, hence docker-in-docker (which is docker-by-docker), won't work. And when I run it as root, other side effects creep into our Jenkins system. – gregturn Nov 30 '20 at 17:10
  • 7
    [Jérôme Petazzoni](https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/) changed his opinion (2020) because there are new tools like [sysbox](https://github.com/nestybox/sysbox) that make it more convenient. He updated his blog post linked in the answer reflecting this. – volkit Dec 07 '20 at 15:34
76

I answered a similar question before on how to run a Docker container inside Docker.

To run docker inside docker is definitely possible. The main thing is that you run the outer container with extra privileges (starting with --privileged=true) and then install docker in that container.

Check this blog post for more info: Docker-in-Docker.

One potential use case for this is described in this entry. The blog describes how to build docker containers within a Jenkins docker container.

However, Docker inside Docker it is not the recommended approach to solve this type of problems. Instead, the recommended approach is to create "sibling" containers as described in this post

So, running Docker inside Docker was by many considered as a good type of solution for this type of problems. Now, the trend is to use "sibling" containers instead. See the answer by @predmijat on this page for more info.

Community
  • 1
  • 1
wassgren
  • 18,651
  • 6
  • 63
  • 77
20

It's OK to run Docker-in-Docker (DinD) and in fact Docker (the company) has an official DinD image for this.

The caveat however is that it requires a privileged container, which depending on your security needs may not be a viable alternative.

The alternative solution of running Docker using sibling containers (aka Docker-out-of-Docker or DooD) does not require a privileged container, but has a few drawbacks that stem from the fact that you are launching the container from within a context that is different from that one in which it's running (i.e., you launch the container from within a container, yet it's running at the host's level, not inside the container).

I wrote a blog describing the pros/cons of DinD vs DooD here.

Having said this, Nestybox (a startup I just founded) is working on a solution that runs true Docker-in-Docker securely (without using privileged containers). You can check it out at www.nestybox.com.

ctalledo
  • 341
  • 3
  • 5
9

Yes, we can run docker in docker, we'll need to attach the unix socket /var/run/docker.sock on which the docker daemon listens by default as volume to the parent docker using -v /var/run/docker.sock:/var/run/docker.sock. Sometimes, permissions issues may arise for docker daemon socket for which you can write sudo chmod 757 /var/run/docker.sock.

And also it would require to run the docker in privileged mode, so the commands would be:

sudo chmod 757 /var/run/docker.sock

docker run --privileged=true -v /var/run/docker.sock:/var/run/docker.sock -it ...
Eduardo Cuomo
  • 17,828
  • 6
  • 117
  • 94
Renu Saini
  • 91
  • 1
  • 3
0

I was trying my best to run containers within containers just like you for the past few days. Wasted many hours. So far most of the people advise me to do stuff like using the docker's DIND image which is not applicable for my case, as I need the main container to be Ubuntu OS, or to run some privilege command and map the daemon socket into container. (Which never ever works for me)

The solution I found was to use Nestybox on my Ubuntu 20.04 system and it works best. Its also extremely simple to execute, provided your local system is ubuntu (which they support best), as the container runtime are specifically deigned for such application. It also has the most flexible options. The free edition of Nestybox is perhaps the best method as of Nov 2022. Highly recommends you to try it without bothering all the tedious setup other people suggest. They have many pre-constructed solutions to address such specific needs with a simple command line.

The Nestybox provide special runtime environment for newly created docker container, they also provides some ubuntu/common OS images with docker and systemd in built. Their goal is to make the main container function exactly the same as a virtual machine securely. You can literally ssh into your ubuntu main container as well without the ability to access anything in the main machine. From your main container you may create all kinds of containers like a normal local system does. That systemd is very important for you to setup docker conveniently inside the container.

One simple common command to execute sysbox:

    dock run --runtime=sysbox-runc -it any_image

If you think thats what you are looking for, you can find out more at their github: https://github.com/nestybox/sysbox

Quicklink to instruction on how to deploy a simple sysbox runtime environment container: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/README.md

gway
  • 31
  • 3