0

The scenario I most feared has happened: my data-only docker container is suddenly empty.

This is not serious: it's a development machine and I have back-up. But I fear this most because I know that I still have holes in my understanding of Docker.

I have read in this answer the following:

Docker containers will persist on disk until they are explicitly deleted with docker rm.

Here are the containers I'm interested in (from a docker ps command):

CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                         PORTS               NAMES
478e59ecd218        dockerlocal_mongo_instance    "/entrypoint.sh mongo"   About an hour ago   Exited (137) 12 minutes ago                        dockerlocal_mongo_instance_1
0ca49f6629cb        tianon/true                   "/true"                  3 hours ago         Exited (0) About an hour ago                       dockerlocal_mongo_data_1

I have a 1) a mongo container which references the data-only container, and 2) the data-only container itself. I recently ran docker rm a couple of times on the mongo dockerlocal_mongo_instance_1 container which references the data-only container.

I can see from the output of the docker ps command (see above) that it says that the data-only container was created '3 hours ago'. But I created it about 2 weeks ago. Somehow my original one has gone. My question is how could this happen? What other possibilities are there?

I have checked my bash command history and the docker rm command was run only on the mongo container, not on the data-only container - which for obvious reasons I have been extremely careful not to touch.

Can anyone shed any light on this? I must have misunderstood something fundamental here.

I would be grateful for any other possible scenarios that could cause the data-container to be trashed and re-created in this way.

Docker compose .yml file (relevant bits):

mongo_data:
    image: tianon/true
    volumes:
        - /data/db
mongo_instance:
    build: mongodb
    volumes_from:
        - mongo_data
    ports:
        - "27017:27017"
    environment:
        - MONGODB_USER=$S_USER_NAME
        - MONGODB_PASS=$S_USER_PASSWORD
#    command: --auth
Community
  • 1
  • 1
mwarren
  • 2,409
  • 1
  • 22
  • 28
  • I don't really understand - what do you mean "the data-only container was re-created"? What recreated it? Are you using Docker Compose? – Adrian Mouat Dec 23 '15 at 17:54
  • That's the whole point, I don't know what recreated it either. But if you look at the CREATED column, it says it was created '3 hours ago'. Whereas I know it was created two weeks ago. What could have recreated it, that's the question. I will edit the question to make this clearer. – mwarren Dec 23 '15 at 17:56
  • @Adrian Mouat Yes, I am using Docker Compose – mwarren Dec 23 '15 at 18:03
  • 1
    Did you try `docker volume ls` (using Docker 1.9+)? That command would list all your volumes, even those without connected container. You'll find that volumes actually reside in `/var/lib/docker/` and you should find your old data there. – gesellix Dec 23 '15 at 19:20
  • @gesellix Thanks for the idea, but there are about 100 volumes, of them all with names like this: 6054b0b4cc561738787e87f84291d06a6cb038120f548229328779f2d7296f16 so I can't make much sense of them. I'm on mac, so /var/lib/docker doesn't exist. I think stuff is in **/Users/username/.docker** on mac, but can't make much out of what I've found there. – mwarren Dec 23 '15 at 20:11
  • 1
    Can you share your Compose file? If you have defined the data container in the Compose file, that's your problem. – Adrian Mouat Dec 23 '15 at 20:22
  • @Adrian Mouat I've added Docker Compose .yml – mwarren Dec 23 '15 at 20:43

3 Answers3

1

There's a couple of things you need to understand.

  • A data container doesn't need to be, and shouldn't be, running. It's really just a namespace for a set of volumes that can the be referred to from other containers. In your case, every time you start up your application, the data container will start, run true, then shut down. It would be better if you just created the container once and never ran it again.
  • Docker Compose defines the running services that make up your application. It has a lot of logic related to deciding when to recreate containers or reuse existing ones, which at some stage has decided to recreate your data container (I'm not sure why in this case). You should only put stuff in Compose that does not need to be persisted. Also note that Compose will attempt to copy volumes from old containers to new ones, which can cause confusion if you're not expecting it.

In your case, the solution is to define the data container outside Compose e.g:

docker run --name mongo_data mongodb echo "Data Container"

This will run the echo command then immediately exit. You can then remove the mongo_data entry from the Compose yaml. Note that I have intentionally used the mongodb image rather than tianon/true; as a data container isn't left running, it won't take up any extra space and using the mongodb image ensures file permissions etc are correct.

Adrian Mouat
  • 44,585
  • 16
  • 110
  • 102
  • Thanks, this is what I was looking for. I shall keep the data container outside of Docker Compose. You have also confirmed what I have just read somewhere that you don't need to use tiny images like tianon/true, you can just reuse existing ones. In fact I read somewhere that tianon can give permissions problems. For the record, my data container was just trashed again - I think from now on I will be able to sleep more soundly. – mwarren Dec 24 '15 at 11:37
  • The other thing you could do is explicitly define a directory for the volume with -v when starting the docker container (e.g. -v /opt/mongo_data:/mongodb), that way you can easily recreate a new container with the same data. – Adrian Mouat Dec 24 '15 at 11:42
  • Sounds great. Could you try to explain to a bear of little brain the difference between my volume declaration, which presumably just creates /data/db on the host and yours? - Cant get my head round it. – mwarren Dec 24 '15 at 11:56
  • The difference is that if you don't specify a host directory, Docker will allocate a directory that it will manage itself for the purpose, normally under `/var/lib/docker`. – Adrian Mouat Dec 24 '15 at 12:37
  • In the end I have decided to drop using a data container. One reason is that I want to keep everything in the .yml file. However I am still a little doubtful about the security of my data. Time will tell. – mwarren Dec 27 '15 at 23:11
0

If you ever ran docker-compose rm (or, worse docker-compose rm -f), that would have deleted all extant containers defined in your docker-compose.yml. Note that, even if you only meant to start the mongo_instance container with docker-compose up mongo_instance, the mongo_data container would have been created as well, as mongo_instance depends on mongo_data, and so doing docker-compose rm while mongo_instance was around would delete both containers.

jwodder
  • 54,758
  • 12
  • 108
  • 124
  • Thanks for your answer, but I can exclude that I ever ran 'docker-compose rm -f', because I was unaware of the -f parameter. I did remove the mongo instance three times today - I know this because I have checked every command in my bash history. But removing the mongo instance should not remove the data container under any circumstances. And 'docker-compose up' has always been run against all the containers in the docker-machine – mwarren Dec 23 '15 at 22:46
0

The answer for me in the end is to forget using a docker data container and just set a normal volume on the mongo container.

mongo_instance:
    build: mongodb
    volume:
        - /data/db:/data/db
    ports:
        - "27017:27017"
    environment:
        - MONGODB_USER=$S_USER_NAME
        - MONGODB_PASS=$S_USER_PASSWORD

Why?

  1. The main reason to use Docker Compose in my view is because of its portability. You easily run all your docker commands in one place, which means a simple installation. So if using a docker data container means moving the creation of the data container out of the .yml file I am complicating everything - I first have to create the data container and then run docker compose: I would much prefer to just upload my .yml file and run it.
  2. I found problems anyway with the creation of the data container on its own before calling docker-compose. You are advised to reuse your own images when creating data containers rather than using small third-party images like tianon/true. But my images, which I might be able to use, only get created when I run docker-compose and I haven't run it yet - a chicken and egg situation.
  3. I tried creating a data container using tianon/true (and others) and I always ran into permission problems.

So I have removed the data container and used a volume parameter on the mongo_instance. I hope this also solves my original problem ...

mwarren
  • 2,409
  • 1
  • 22
  • 28