How to record the peak memory usage of a Docker container?

Question

I'm running a number of short-lived docker containers each of which does some memory-intensive batch processing. I'm looking for a way to find the peak memory usage each container hit while it was running. Knowing this will allow me to optimize the infrastructure I run these containers on for future runs.

One naive way to achieve this is redirecting the streaming output of docker stats to some file: docker stats container_id > stats.log. However, this requires running a process for each container and then sorting through very verbose logs to find the peak usage. I'm wondering if there's not an easier way.

score 15 · Accepted Answer · answered Oct 08 '19 at 10:12

If you are interested in the process with PID=1 inside the container, you can find the PID this process has on the host and then use:

grep VmPeak /proc/$PID/status

Example with a mongo container:

This container has a single process:

$ docker container exec -it mongo top -bn 1
top - 10:04:51 up 32 min,  0 users,  load average: 0.36, 0.52, 0.55
Tasks:   2 total,   1 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.1 us,  2.0 sy,  0.3 ni, 90.6 id,  0.7 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :  6103572 total,  2642744 free,  1352032 used,  2108796 buff/cache
KiB Swap:  1942896 total,  1942896 free,        0 used.  4277928 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   64 root      20   0   38624   3116   2724 R   6.7  0.1   0:00.89 top
    1 mongodb   20   0 1094540  80100  35916 S   0.0  1.3   0:22.51 mongod

To get the PID of this process from the host's perspective:

$ docker inspect -f '{{.State.Pid}}' mongo
2532

and finally:

$ grep VmPeak /proc/2532/status
VmPeak:  1094540 kB

Links:

`grep: /proc/36133/status: No such file or directory` on macOS — Nakilon, Apr 16 '21 at 10:20
Looks like on macOS you can obtain this data by reading the /proc thing inside the container. — Nakilon, Apr 16 '21 at 10:32

score 3 · Answer 2 · answered Jan 14 '22 at 13:49

docker stats has the --format option to customize the output. For example we can output only memory usage:

 docker stats --no-stream --format '{{.MemUsage}}' CONTAINER_ID 
516KiB / 8GiB

CONTAINER_ID in the command must be replaced with the ID of our container (which we can get with docker ps).

516KiB is the current memory usage of the container and 8GiB is the maximum memory that the container is allowed to use.

On GNU/Linux, we can filter out the memory usage from the output with cut and redirect the output into a file:

docker stats --no-stream --format '{{.MemUsage}}' CONTAINER_ID | cut -d '/' -f 1 >>docker-stats

Finally, we can put this command into a while loop and add a one second delay between iterations:

while true; do docker stats --no-stream --format '{{.MemUsage}}' CONTAINER_ID | cut -d '/' -f 1 >>docker-stats; sleep 1; done

This way, in the docker-stats file, we will get a list of memory usage values of the container with one second resolution. Of course, if a peak lasts less than one second we may miss it.

Docker stats docs.

How to record the peak memory usage of a Docker container?

2 Answers2