We are deploying a stack, the compose file list two services:
We saw that the mysql Dockerfile has a VOLUME directive, to persist the databases to a Docker volume
Yet, if we docker stack rm then docker stack deploy our compose file, we lose all the database content.
Is that the expected behaviour? What would be the rationale for it?
Swarm mode doesn't build docker images, so it's safe to say that docker swarm is ignoring every line of your Dockerfile. The resulting image will have a volume identified which will be created when the container is run, whether that is inside or outside of swarm mode. If you do not specify a volume mount at runtime, docker will give you an anonymous volume at that location, which is difficult to identify later on, and local to the node where the container is running. With a new stack, the previous anonymous volume won't be used, and in various situations, the anonymous volume can be automatically deleted (e.g. when containers are configured to automatically delete on exit, which I'm not sure if it applies to swarm mode).
A named volume can make the data easier to reused in a single node cluster. When you get to a multi-node cluster, you need to move this data off of the node where the container is running. For details on how to use something like NFS for external data storage, see this answer to a related question.
The Dockerfile has a volume but you probably haven't mounted that volume as accessible outside of that container, which means it probably isn't being pulled out to the persistent overlay storage to be shared between containers/container deployments.
In your database service in the docker-compose.yml file you're going to need a volumes section. I'm omitting your current setup since you didn't provide it and I'm going to replace it with ellipses ....
mySQL:
...
volumes:
- mySQL-data:/var/lib/mysql
This tells the system that you want to persist the /var/lib/mysql directory outside of the container in your shared overlay that I randomly named mySQL-data.
Being in swarm mode is not relevant in this case.
If you are only relying on the anonymous volume defined in the Dockerfile, running a new container will create and mount a new fresh volume. You need to specifically mount a named volume at container start (in your case add it in your compose file) to remount the same volume between runs.
If you need to remount a lost data volume, it might still be possible if you did not prune data on your server. You will just need to find the relevant volume (that has a hash as name), eventually rename it and remount it in your new container.
I played the following scenario to illustrate my point:
First get the image and have a look:
$ docker pull mysql:latest
$ docker image inspect mysql:latest
From this last command we can see there is a volume declared for /var/lib/mysql
I'm on my dev machine. Cleaned up everything so I have no volumes at this time
$ docker volume ls
DRIVER VOLUME NAME
$
Start a container then look at volumes again
$ docker run -d --name test -e MYSQL_ALLOW_EMPTY_PASSWORD=1 mysql:latest
37a92341f52b189d00636d1f03ecfbd4e3e7e5d55b685f5ec254971d7732566c
$ docker volume ls
DRIVER VOLUME NAME
local 50f9810d13271c7c91b7e025e139db480f288a936f0a55f85d9580ef3aa83af7
Now add some data to the container
$ docker exec -it test bash
root@37a92341f52b:/# mysql
Welcome to the MySQL monitor. [... snip ...]
mysql> create database testso;
Query OK, 1 row affected (0.03 sec)
mysql> use testso;
Database changed
mysql> create table test (id int not null primary key);
Query OK, 0 rows affected (0.08 sec)
mysql> insert into test values (1);
Query OK, 1 row affected (0.05 sec)
mysql> select * from test;
+----+
| id |
+----+
| 1 |
+----+
1 row in set (0.00 sec)
mysql> exit
Bye
root@37a92341f52b:/# exit
$
Create a new container
$ docker rm -f test
test
$ docker run -d --name test -e MYSQL_ALLOW_EMPTY_PASSWORD=1 mysql:latest
79148de09d7a3e13db338da133cfd7d44fe3590dc1c7ffe6129722c5c6baea21
$ docker volume ls
DRIVER VOLUME NAME
local 50f9810d13271c7c91b7e025e139db480f288a936f0a55f85d9580ef3aa83af7
local ef6fb2647c11c10ef98d25ca0dc2bd43231729ed18386c65768a0ad808fca93b
As you can see we have a second volume for the new container. I will not show this here but if I connect, the data is empty as in your case. Now let's try to recover.
First a little cleanup
$ docker rm -f test
test
$ docker volume rm ef6fb2647c11c10ef98d25ca0dc2bd43231729ed18386c65768a0ad808fca93b
ef6fb2647c11c10ef98d25ca0dc2bd43231729ed18386c65768a0ad808fca93b
We want to have a human readable name but we cannot rename a volume. What I did is mount the old one and a new named volume in a busybox container to transfer the data over.
$ docker run -it --rm -v 50f9810d13271c7c91b7e025e139db480f288a936f0a55f85d9580ef3aa83af7:/mysql/old -v mysql_data:/mysql/new busybox:latest
/ # cd /mysql/
/mysql # mv old/* new/
/mysql # exit
We now have this new volume and we can get rid off the anonymous one
$ docker volume ls
DRIVER VOLUME NAME
local 50f9810d13271c7c91b7e025e139db480f288a936f0a55f85d9580ef3aa83af7
local mysql_data
$ docker volume rm 50f9810d13271c7c91b7e025e139db480f288a936f0a55f85d9580ef3aa83af7
50f9810d13271c7c91b7e025e139db480f288a936f0a55f85d9580ef3aa83af7
And finally we remount the named volume in a fresh mysql container to get our data back
$ docker run -d --name test -e MYSQL_ALLOW_EMPTY_PASSWORD=1 -v mysql_data:/var/lib/mysql mysql:latest
3f6eff0b7660f3f8e9518564affc6555acb17184845156099d18300b3e76f4a2
$ docker exec -it test bash
root@3f6eff0b7660:/# mysql
Welcome to the MySQL monitor. [... snip ...]
mysql> use testso
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from test;
+----+
| id |
+----+
| 1 |
+----+
1 row in set (0.01 sec)
