11

I have multiple docker processes running on a machine, I'd like to monitor them and restart docker process once it passes certain memory threshold. Can I make docker restart it's process when it hits memory threshold? What's my options?

Thanks

Alex

AlexV
  • 3,836
  • 7
  • 31
  • 37

2 Answers2

13

You could make a shell script to monitor resources usage, and restart Docker daemon when it reaches your memory limit, but I think that's not actually a good approach.

Using this command you can see your containers ordered by memory usage. Find which container is using too much memory and try to find the reason because that's happening.

docker stats --no-stream --format "table {{.Name}}\t{{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}" | sort -k 4 -h

Also, if memory consumption of your containers is normal, but you want to limit it, you can limit resources assigned to each container. You can do this using option --memory in docker run.

For further information about memory limits, check this info in Docker docs: https://docs.docker.com/engine/admin/resource_constraints/

Hope this helps, good luck.

Edit: Answering your response, if your container runs out of memory, it will be automatically killed by the kernel. You can configure a memory limit using option --memory and set restart policy as --restart=always. This way, your container will be killed automatically by an OOM (out-of-memory) error, but it will be restarted since its restart policy is to keep restarting after any error.

kstromeiraos
  • 4,659
  • 21
  • 26
  • I have a long running `python` process inside `docker` and over time it consumes more and more memory, without releasing it. (https://stackoverflow.com/a/1316799/248723) I want to restart it occasionally. If I pass `--memory` to `docker run` what will `docker` do when the process inside container runs out of memory? – AlexV Jun 21 '17 at 11:45
  • 1
    Check my edit, I think it should solve your problem. Please, validate the answer if that happened :) – kstromeiraos Jun 21 '17 at 12:19
  • Did you try this? Because I tried limit memory by using --memory, but it will never be killed by OOM, just stick at the max memory. I need to kill the container because in the max memory the performance is really bad. –  Feb 04 '21 at 12:15
9

Always remember that if you don't set --memory-swap, the container will no to restart at --memory value limit because Docker will use Swap after reach --memory limit, so, if you want to restart at --memory value, you need to set --memory-swap with the same value as --memory limit.

docker run --memory 50m --memory-swap 50m --rm -it progrium/stress --vm 1 --vm-bytes 62914560 --timeout 1s
Alejandro Mendez
  • 91
  • 1
  • 2