docker-compose restart connection pool full

Question

My team and I are converting some of our infrastructure to docker using docker-compose. Everything appears to be working great the only issue I have is doing a restart it gives me a connection pool is full error. I am trying to figure out what is causing this. If I remove 2 containers or (1 complete setup) it works fine.

A little background on what I am trying to do. This is a Ruby on Rails application that is being ran with multiple different configurations for different teams within an organization. In total the server is running 14 different containers. The host server OS is CentOS, and the compose command is being ran from a MacBook Pro on the same network. I have also tried this with a boot2docker VM with the same result.

Here is the verbose output from the command (using the boot2docker vm) https://gist.github.com/rebelweb/5e6dfe34ec3e8dbb8f02c0755991ef11

Any help or pointers is appreciated.

score 15 · Answer 1 · answered Mar 20 '19 at 03:36

I have been struggling with this error message as well with my development environment that uses more than ten containers executed through docker-compose.

WARNING: Connection pool is full, discarding connection: localhost

I think I've discovered the root cause of this issue. The python library requests maintains a pool of HTTP connections that the docker library uses to talk to the docker API and, presumably, the containers themselves. It's my hypothesis that only those of us that use docker-compose with more than 10 containers will ever see this. The problem is twofold.

requests defaults its connection pool size to 10, and
there doesn't appear to be any way to inject a bigger pool size from the docker-compose or docker libraries

I hacked together a solution. My libraries for requests were located in ~/.local/lib/python2.7/site-packages. I found requests/adapters.py and changed DEFAULT_POOLSIZE from 10 to 1000.

This is not a production solution, is pretty obscure, and will not survive a package upgrade.

Related: [Can I change the connection pool size for Python's “requests” module?](https://stackoverflow.com/q/18466079/55075) — kenorb, Mar 25 '19 at 21:15
So it is a docker issue, do you know if it has been addressed in most recent versions or if there is an issue open for this, I could not find anything about — Glabler, Jan 22 '20 at 14:33

score 2 · Answer 2 · answered Apr 04 '18 at 08:52

2

You can try reset network pool before deploy

$ docker network prune

Docks here: https://docs.docker.com/engine/reference/commandline/network_prune/

answered Apr 04 '18 at 08:52

andriy-baran

620
4
16

1

Thank you for the response, this unfortunately did resolve the issue. It also is never the same container, it runs the same code but not the same specific instance. – Ryan Condron Apr 04 '18 at 14:21

score 2 · Answer 3 · edited Sep 16 '21 at 18:30

2

I got the same issue with my Django Application. Running about 70 containers in docker-compose. This post helped me since it seems that prune is needed after setting COMPOSE_PARALLEL_LIMIT

I did:

docker-compose down
export COMPOSE_PARALLEL_LIMIT=1000
docker network prune
docker-compose up -d

edited Sep 16 '21 at 18:30

swiss_knight

5,787
8
50
92

answered Apr 28 '20 at 14:22

Jimmy Engelbrecht

653
6
9

score 1 · Answer 4 · answered May 07 '20 at 06:51

For future readers. A small addition to the answer by @andriy-baran You need to stop all containers, delete them and them run network prune (because the prune command removes unused networks only)

So something like this:

docker kill $(docker ps -q)
docker rm $(docker ps -a -q)
docker network prune

docker-compose restart connection pool full

4 Answers4

Linked