Docker Network not Found

Question

In our team, we are currently transitioning to Docker to deploy everything on our server.

We are using Docker Swarm and multiple (10+) compose files defining plenty (20+) of services. Everything works beautifully so far, except when we take down our stack using docker stack rm <name> (and redeploy using docker stack deploy <options> <name>): about every second time, we get the following error:

Failed to remove network <id>: Error response from daemon: network <id> not foundFailed to remove some resources from stack: <name>

When using docker network ls, the network is indeed not removed, however, docker network rm <id> always results in the following:

Error response from daemon: network <id> not found

What makes that even more strange is the fact that docker network inspect <id> returns a normal output. The networks are always overlay networks that are created with the compose files used to deploy our stack. Currently, we only have a single node in our Swarm.

Our current "workaround" is to restart Docker (which resolves the issue), but that is not a viable solution in a production environment. Leaving the swarm and joining it again does not resolve the issue either.

At first, we thought that this issue is related to Docker for Mac only (as we first encountered the issue on local machines), however, the same issue arises on Debian Stretch. In both cases, we use the latest Docker distribution available.

I would really appreciate any help!

Did you check to see if all the containers using this network are not running? the stack rm issues the command to delete all the services immediately but the containers themselves takes a bit of time. Also, are the networks has any configurations? IPAM, specific drivers? — nadavbrkt, Nov 24 '18 at 16:57

score 260 · Answer 1 · answered Mar 27 '19 at 18:01

260

If you are attempting to add a container to an existing network that no longer exists, then you can use docker-compose up --force-recreate. I found this GitHub issues comment to be a helpful overview.

answered Mar 27 '19 at 18:01

Nick Beaird

2,701
1
9
5

2

This did the trick for me! – Vladislavs Dovgalecs May 19 '22 at 21:25
1

This did not resolve the issue on Docker version 20.10.14, build a224086. – Yuki Matsukura May 27 '22 at 15:05

score 31 · Answer 2 · edited Nov 27 '18 at 10:46

31

You can always use docker system prune -a to get rid of the old network. This will not delete your volumes.
It will take longer to docker-compose up --build -d the next time, but it will get you past your current problem.

edited Nov 27 '18 at 10:46

JoSSte

2,953
6
34
54

answered Nov 27 '18 at 01:58

fcnorman

1,154
9
19

35

`docker system prune -a` will remove all your existing images and containers. DO NOT USE! – fnagel Nov 26 '20 at 14:27
1

Indeed. Not sure what the additional `-a` is usefull for, because it does the same without it. – bomben Dec 02 '20 at 13:04
My problem fixed without -a flag – Viljami Apr 11 '22 at 08:07
`-a` also removes tagged images which are not in use. https://docs.docker.com/config/pruning/#prune-images – Ryan Apr 28 '22 at 07:31

score 19 · Answer 3 · edited Mar 25 '20 at 05:53

After using the docker prune command, I was unable to launch the docker container on a network.

The following errors stated:

ERROR: for jekyll-serve Cannot start service jekyll-serve: network b52287167caf352c7a03c4e924aaf7d78e2bc372c703560c003acc758c013432 not found ERROR: Encountered errors while bring up the project.

docker system prune enabled me to begin using docker-compose up again.

More info here: https://docs.docker.com/config/pruning/

score 12 · Answer 4 · answered Mar 10 '21 at 14:09

12

Deleting the "network not found" in docker

Inspect the network which we are unable to delete

docker network inspect <id> or <name>

Disconnect the network

docker network disconnect -f <networkID> <endpointName> or <endpointId>

Delete unused networks

docker network prune

answered Mar 10 '21 at 14:09

Jinna Balu

6,747
38
47

1

someone have tested? – mik3fly-4steri5k Mar 18 '21 at 12:53
2

This should be the accepted answer. I automated this solution and trigger it each time i have a ghost network. Working very well. – mik3fly-4steri5k Mar 18 '21 at 13:14
1

to get this answer I have spent almost 4 hours finding out why it will occur and how to solve it. – Jinna Balu Mar 18 '21 at 16:40
1

I can confirm that this works. – Aníbal Rivero May 26 '22 at 22:03
Didn't work in my case, unfortunatelly – PlayHardGoPro Aug 24 '23 at 17:31

score 7 · Answer 5 · answered Nov 21 '18 at 17:59

7

That sounds exactly like this issue.

Stack rm followed "too fast" by stack deploy would race for the creation/removal of networks, possibly other stack resources.

The issue is still open as of today (docker/cli), but you could try the workaround suggested:

until [ -z "$(docker service ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
  sleep 1;
done

until [ -z "$(docker network ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
  sleep 1;
done

answered Nov 21 '18 at 17:59

Julio Daniel Reyes

5,489
1
19
23

Unfortunately, this does not solve the problem .. when I use this code in a script, it is stuck in an infinite loop as (at least) one network is _never_ removed. – borchero Nov 21 '18 at 20:08
Well... [here](https://github.com/moby/moby/issues/29293) is the same issue again, still open. They also suggested the restart of the daemon, did you do that before testing the script? – Julio Daniel Reyes Nov 21 '18 at 20:37
Also it says it takes a while to actually delete the network, can you try running it for a couple of minutes? – Julio Daniel Reyes Nov 21 '18 at 20:39
Yeah, as indicated in my question, restarting the daemon is my "hardfix", but it really seems kind of strange. – borchero Nov 21 '18 at 20:40
3

It ran for about 10 minutes and did not complete. When I run `docker network ls`, one of the networks is still visible, over 30 minutes after removing the stack. But I guess there seems to be no solution to the problem at the moment (based on the issues that you linked). – borchero Nov 21 '18 at 20:41

score 7 · Answer 6 · answered Apr 16 '21 at 02:10

7

I could not get rid of the networks by any of the methods in previous answers.

This is what worked for me.

systemctl restart docker

answered Apr 16 '21 at 02:10

Ashim

411
1
8
17

This workaround was in the original question BTW, albeit less explicitly. – Christian Jan 06 '22 at 16:31

score 3 · Answer 7 · answered Mar 29 '23 at 08:40

To me it helped to use

docker network connect existingNetwork container

for all network failing containers, nothing else seemed to help to me, since they seemed to be connected as implied to a non existing network, here mentioned disconnecting them dindt help only re/connecting to existing, and removing network that non-existing ended up in an error "could not remove a non existing network" ofc

score 1 · Answer 8 · edited Apr 28 '23 at 13:31

1

None of the above worked for me. I've used the following docker-compose command to fix.

docker-compose up --force-recreate --remove-orphans

edited Apr 28 '23 at 13:31

Aki

2,818
1
15
23

answered Apr 25 '23 at 08:51

Phillip Dawson

11
1

score 0 · Answer 9 · answered Mar 02 '22 at 23:41

This is the experience I got and I think it might help. Docker network is capable of doing bridging. In course told, a container can disconnect and connect from one to the other. If one disconnects from current and connect to the other, and the current disappear due to shutdown/network prune, the independent container will lose the connection. Later, when you try to start, only found "network not found" error.

The solution to this is start swarm/cluster (in my case I start with docker-compose up), disconnect the container (even it's yet up) from that network using force (-f). Connect back to that (different ID, but same name) network. Now you can successfully start it without "network not found" error. So, the point is it maybe happens to see same name and different ID network.

score -1 · Answer 10 · answered Jun 19 '19 at 13:42

-1

old containers are still using old network. Probably you removed networks but forgot to rm old containers. Just remove old containers, create your network and build again.

answered Jun 19 '19 at 13:42

joaoluiznaufel

27
3

Docker Network not Found

10 Answers10

Deleting the "network not found" in docker