Docker container keeps growing

Question

I have a python script that on a loop

Downloads video chunks from AWS S3 to /filename.
Sorts files in order and concatenates them.
Uploads entire processed video file to AWS S3.
Deletes folder /filename.

Then continues on a loop until the AWS SQS queue is empty.

Script works great! I have run it for months. The hard drive space varies but never gets about 5%, depending on size of the video.

I decided to put this script in a docker container and run docker-compose so I could run a bunch of them at a time.

The problem is the hard drive fills up! I know with 5 running the space on the disk will be higher, but when I'm done processing, the file gets deleted.

But with docker, it seems to be a cache or something. I exec into each container and they are running fine. Deleteing old files and all.

No clue what the difference between, in a docker container and running as a service would have as impact on the HD.

To add to this, when I "rm" the docker containers, the hard drive space frees up. I run a "docker ps -s" and the space on the containers is not crazy. Just seems like when you "rm" a file inside the docker container it never really "rm"s it.

Ray · Answer 1 · 2016-05-01T18:02:29.083

7

If you're downloading the image to a directory NOT volumed mapped from the host, the docker container will not release the used disk space until the container is removed--anything done in the container is ephemeral, but the HOST doesn't know the state of what's going on inside the container.

In this sense it's a lot like a virtual machine image, backed by a file that just grows as needed, but never shrinks. Docker has a directory for a running container tracking changes. On the host you can find the files backing the running container in /var/lib/docker/containers/<id>

If you need your containers to share disk space, I'd recommend you map a shared volume from the host into each docker container images to share.

Try the following

 docker run -ti -v /host/dir:/container/dir ubuntu bash

The above would run the ubuntu image in terminal interactive mode and mounting the host's directory /host/dir inside the running container. Anything the container writes to /container/dir will appear in the hosts /host/dir and any other containers mounting it will see the changes as well.

Just remember anything done in the shared volume is seen by all containers that mount it, so be careful when adding and deleting files/directories from it!

edited May 01 '16 at 18:02

answered May 01 '16 at 17:14

Ray

40,256
21
101
138

So essentially it is kind of useless to even run a "rm filename" inside the container, because it never really releases the disk space. How does a nginx or any web server container not fill up with temp cache all the time? – joeymiller May 01 '16 at 18:04
2

@user2457010 you can mount the host `/tmp` directory. Fun fact, even when you `rm` things the container disk usage grows! Docker keeps everything as a log of changes to the container's image. Play around looking at the `/var/lib/docker/container//-json.log` file. I try to treat containers internal file system as read only as much as possible, using mounted volumes when needed. – Ray May 01 '16 at 18:24
I totally agreed with what Ray said above, in my case the ever-growing log file `/var/lib/docker/containers//-json.log` was the main problem. It keeps stacking up all the standard console out. I resolved the ever-growing log file issue by limiting the size with setting "log-opts" option in `/etc/docker/daemon.json` file. please check this link for the setup detail https://docs.docker.com/config/containers/logging/json-file/ – Steve Park Oct 14 '19 at 01:36

score 2 · Answer 2 · answered May 01 '16 at 17:15

2

I would suggest you to use volumes, and mount these volumes in your containers. Changes on volumes are instantaneous, as opposed to changes made to the containers filesystem (which is not really removed until you delete the container).

Have a look at the docs here

answered May 01 '16 at 17:15

JustDanyul

13,813
7
53
71

So a named volume will release used disk space? So adding and removing files with a named volume is the way to go? – joeymiller May 01 '16 at 18:06
1

Yes, if you need to manipulate data on the filesystem, use mounted volumes. – JustDanyul May 01 '16 at 18:53

score 1 · Answer 3 · answered Jan 19 '22 at 15:21

Two things to look at:

The container's read/write filesystem layer. You can get an idea for what is happening here with docker diff on the specific container id to see all the filesystem changes made by your container.
The container logs. You can view them with docker logs. On a long running container, these may become excessive. See this answer for the process to limit the log size on one container or to make those limits the defaults for all newly created containers.

Lastly, if you are running docker inside a VM, e.g. Docker Desktop, then you may be seeing filesystem changes expanding the copy-on-write VM filesystem which isn't easy to undo. (The underlying device doesn't know the OS isn't using the bytes anymore, so once the zero'd bytes have been replaced by data, even if the FS no longer points to those bytes, they are assumed in use.) However, since you indicated removing the container frees the space, this is unlikely to be your issue.

It's also unlikely to be solved by a volume, since your application makes the same filesystem changes to the volume that it would to the read-write layer of the container. All it would do is persist those changes between containers.

Docker container keeps growing

3 Answers3

Linked