I'd like to mount a host directory in docker that on the outside is actually read/only. But I'd like it to appear inside the container as read/write.
So that files/directories can be written to but not changed on the outside. Is this possible using some kind of overlay process?
You can do this without running privileged containers, and without any other 3rd party tools, using the local volume driver. The local volume driver will pass any options to the mount syscall, so anything you can do with mount you can do as a volume in docker. The only prerequisite is that you create the overlay directories in advance and clean them up yourself.
First, lets create the directories and some read only data:
$ mkdir -p {ro-data,upper1,upper2,upper3,work1,work2,work3}
$ ls
ro-data upper1 upper2 upper3 work1 work2 work3
$ vi ro-data/data.txt
$ cat ro-data/data.txt
This is a data file.
It should be read-only on the host upper dir.
Next, lets create a named volume with the overlay options and run a container with it:
$ docker volume create --driver local --opt type=overlay \
--opt o=lowerdir=${PWD}/ro-data,upperdir=${PWD}/upper1,workdir=${PWD}/work1 \
--opt device=overlay overlay1
overlay1
$ docker container run -d --rm -v overlay1:/data --name cont1 busybox tail -f /dev/null
a6269cb6c68469aa4f57aae554c5f0823f1103715334b3719c5567abc7d55daa
Then, lets do the same with a --mount option to run, which gets slightly more complicated because of the nested comma separated strings. Escaped quotes works around that:
$ docker run -d --rm \
--mount type=volume,dst=/data,volume-driver=local,volume-opt=type=overlay,\"volume-opt=o=lowerdir=${PWD}/ro-data,upperdir=${PWD}/upper2,workdir=${PWD}/work2\",volume-opt=device=overlay \
--name cont2 busybox tail -f /dev/null
7329ae4ba4046782166b045611ecccb129f5e557df7eb4da95ec9063a0fe234e
Finally, let's us a compose file:
$ vi docker-compose.yml
$ cat docker-compose.yml
version: '3'
volumes:
overlay3:
driver: local
driver_opts:
type: overlay
o: lowerdir=${PWD}/ro-data,upperdir=${PWD}/upper3,workdir=${PWD}/work3
device: overlay
services:
overlay3:
image: busybox
command: tail -f /dev/null
container_name: cont3
volumes:
- overlay3:/data
$ docker-compose up -d
Creating network "vol-overlay_default" with the default driver
Creating volume "vol-overlay_overlay3" with local driver
Creating cont3 ... done
Everything is running, lets verify the data file is there:
$ docker exec cont1 ls -l /data
total 4
-rw-r--r-- 1 1000 1000 67 Nov 8 16:29 data.txt
$ docker exec cont2 ls -l /data
total 4
-rw-r--r-- 1 1000 1000 67 Nov 8 16:29 data.txt
$ docker exec cont3 ls -l /data
total 4
-rw-r--r-- 1 1000 1000 67 Nov 8 16:29 data.txt
Next, we can make some changes to the directory in container 1, and delete the file in container 2:
$ echo "container 1 adds lines" | docker exec -i cont1 tee -a /data/data.txt
container 1 adds lines
$ echo "writing to another file" | docker exec -i cont1 tee -a /data/container1.txt
writing to another file
[11:48:30] [bmitch@bmitch-asusr556l:~/data/docker/test/vol-overlay] [master]
$ docker exec cont2 rm /data/data.txt
Verify each container see's the changes, or lack thereof:
$ docker exec cont1 ls -l /data
total 8
-rw-r--r-- 1 root root 24 Nov 8 16:48 container1.txt
-rw-r--r-- 1 1000 1000 90 Nov 8 16:47 data.txt
$ docker exec cont2 ls -l /data
total 0
$ docker exec cont3 ls -l /data
total 4
-rw-r--r-- 1 1000 1000 67 Nov 8 16:29 data.txt
$ docker exec cont1 cat /data/data.txt
This is a data file.
It should be read-only on the host upper dir.
container 1 adds lines
$ docker exec cont3 cat /data/data.txt
This is a data file.
It should be read-only on the host upper dir.
And show the host directory is unchanged:
$ ls -l ro-data
total 4
-rw-r--r-- 1 bmitch bmitch 67 Nov 8 11:29 data.txt
$ cat ro-data/data.txt
This is a data file.
It should be read-only on the host upper dir.
The changes were all made only to the upper directories:
$ ls -l upper*
upper1:
total 8
-rw-r--r-- 1 root root 24 Nov 8 11:48 container1.txt
-rw-r--r-- 1 bmitch bmitch 90 Nov 8 11:47 data.txt
upper2:
total 0
c--------- 1 root root 0, 0 Nov 8 11:48 data.txt
upper3:
total 0
After removing the containers and volumes, you'll need to manually remove the upper directories. Just as docker doesn't create them for you, it doesn't delete them either, exactly the same as if you ran the mount command yourself.
Edit: Check @javabrett's comment:
Upvoted despite this solution having a sunset. See answer regarding overlay-upperdir-on-overlay being disabled on 4.8 kernels and newer.
See: https://stackoverflow.com/a/50917037/644504
This is what I do:
On the host:
Load the directory as read only.
docker run --privileged -v /path/on/host:/path/on/client-read-only:ro -it ubuntu /bin/bash
On the client:
On the client use OverlayFS over the read-only directory mounted from the host.
mount -t overlayfs none -o lowerdir=/path/on/client-read-only,upperdir=/path/on/client /path/on/client
Then use /path/on/client to read/write the files.
Edit: if you have a 3.18+ kernel on your host, you may prefer using this on the client:
mount -t overlay overlay -o lowerdir=/path/on/client-read-only,upperdir=/path/on/client,workdir=/path/on/client-workdir /path/on/client
Which isn't overlayfs. With overlayfs I had an issue regarding being unable to use rm. overlay solved this problem for me.
Not an option anymore from inside the container (possibly because overlay-over-overlay is disabled in ~ 4.4 kernels)
$ uname -a && \
docker run --privileged --rm debian:latest sh -c "mkdir upper lower work merged && mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged/; dmesg|tail -1"
Linux preprod 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux
mount: wrong fs type, bad option, bad superblock on overlay,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
[288426.860631] overlayfs: filesystem on 'upper' not supported as upperdir
ANYWAY An alternative is to create the overlay on the host and bind it to the guest:
$ mkdir upper lower work merged && \
touch upper/up lower/low && \
sudo mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged/ && \
docker run --rm -v $(pwd)/merged:/tmp/merged debian:latest sh -c "touch /tmp/merged/new-from-container"
$ ls upper/ lower/ merged/
lower/:
low
merged/:
low new-from-container up
upper/:
new-from-container up
This is possible if you avoid setting upperdir to a local dir (ie already an overlay). But you can use tmpfs instead (tested on kernel 4.9):
# On the host to run the container
docker run --cap-add=SYS_ADMIN -i -t -v ~/host-folder-to-mount:/root/folder-ro:ro ubuntu
# Inside the container
# Need to create the upper and work dirs inside a tmpfs.
mkdir -p /tmp/overlay && \
mount -t tmpfs tmpfs /tmp/overlay && \
mkdir -p /tmp/overlay/{upper,work} && \
mkdir -p /root/folder && \
mount -t overlay overlay -o lowerdir=/root/folder-ro,upperdir=/tmp/overlay/upper,workdir=/tmp/overlay/work /root/folder
All credits goes to https://gist.github.com/detunized/7c8fc4c37b49c5475e68ef9574587eee
I would recommend to see whether your filesystem support overlayfs or not; and it can be verified with
$> grep overlayfs /proc/filesystems
$> overlayfs overlay
If so, then I would recommend you to create overlayfs in host machine and mount the merge directory to the Docker container so that you can manage things from the host machine rather than some on host and some on Docker container.
I followed the following steps to achieve this: Let me take an example; I have source-code and I want to build that for multiple platforms like i386, x86_64 and amd64; The source-code will remain same for all platform; where as the executable (.obj and exe) of each platform will differ; so we need executable in each specific platform directory
sudo mount -t overlay overlay -o lowerdir=/home/viswesn/source-code,upperdir=/home/viswesn/i386_executable,workdir=/i386 /home/viswesn/i386_merged
sudo mount -t overlay overlay -o lowerdir=/home/viswesn/source-code,upperdir=/home/viswesn/x86_64_executable,workdir=/x86_64 /home/viswesn/x86_64_merged
It says, any object files or executable created out of source files will remain in /home/viswesn/X_executable directory and source code will remain in /home/viswesn/source-code; where as /home/viswesn/X_merged/ will contain both source-code and executable of specific platform;
Now we should mount the X_merged directory as volume to the Docker container for building the source code for each platform
For i386:
sudo docker run --privileged -v /home/viswesn/i386_merged/:/source-code -it ubuntu-trusty:14:01 /bin/bash
For amd64:
sudo docker run --privileged -v /home/viswesn/amd64_merged/:/source-code -it ubuntu-amd64:14:01 /bin/bash
With this, the same source code is build simultaneously for all the platform parallel'ly without multiple copies of source-code.
