Sudden I/O errors with docker-compose

Question

The project I'm working on has a docker-compose environment for development, and all of a sudden multiple containers start to have I/O errors like this one:

Error: EIO: i/o error, open '/usr/app/src/components/common/FilterComponent.tsx'

NGINX crashes for the same reason, and so on:

nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (5: Input/output error)

Docker has enough disk space (112GB avail/32GB used), everything else on my Mac seems to work fine. The only way to make it work again is a Docker factory reset, until it happens again (~1-2 days).

This is the docker-compose.yml file, pretty basic if you ask me:

x-env-files: &env-files
  env_file:
    - docker.env
version: '3.3'
services:
  mongo-store:
    image: mongo:latest
    volumes:
      - /data/db
    ports:
      - 27017:27017
    networks:
      - backend
    command: ['mongod', '--bind_ip', '0.0.0.0']
  application-ui:
    build:
      context: .
      dockerfile: Dockerfile-dev
    image: application-ui:dev
    ports:
      - 5000:5000
    networks:
      - frontend
    stdin_open: true
    volumes:
      - ./src/:/usr/app/src/
    command: ['npm', 'run', 'start']

  application-service:
    build:
      context: ../application-service
      dockerfile: Dockerfile-local
    image: application-service:dev
    ports:
      - 9000:9000
    networks:
      - backend
      - frontend
    environment:
      - 'GITLAB_TOKEN=${GITLAB_TOKEN}'
    <<: *env-files
    volumes:
      - ../application-service/src/:/usr/app/src/
  nginx:
    image: nginx:latest
    volumes:
      - ./nginx/nginx.local.conf:/etc/nginx/nginx.conf
    ports:
      - 80:80
      - 443:443
    networks:
      - backend
      - frontend
networks:
  backend:
  frontend:

This is the output of docker run --rm -v /:/host busybox df -h:

Filesystem                Size      Used Available Use% Mounted on
overlay                 149.4G     34.8G    106.9G  25% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                     3.9G         0      3.9G   0% /sys/fs/cgroup
shm                      64.0M         0     64.0M   0% /dev/shm
overlay                   3.9G    316.0K      3.9G   0% /host
dev                       3.8G         0      3.8G   0% /host/dev
shm                       3.9G         0      3.9G   0% /host/dev/shm
/dev/vda1               149.4G     34.8G    106.9G  25% /host/etc/cni/net.d
/dev/vda1               149.4G     34.8G    106.9G  25% /host/etc/kubernetes
tmpfs                   796.0M    480.0K    795.5M   0% /host/etc/resolv.conf
tmpfs                   796.0M    480.0K    795.5M   0% /host/run/config
tmpfs                   796.0M    480.0K    795.5M   0% /host/run/desktop
tmpfs                   796.0M    480.0K    795.5M   0% /host/run/guest-services
tmpfs                   796.0M    480.0K    795.5M   0% /host/run/host-services
cgroup_root              10.0M         0     10.0M   0% /host/sys/fs/cgroup
/dev/vda1               149.4G     34.8G    106.9G  25% /host/usr/libexec/kubernetes/kubelet-plugins
/dev/vda1               149.4G     34.8G    106.9G  25% /host/var/lib
/dev/vda1               149.4G     34.8G    106.9G  25% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/docker
tmpfs                     3.9G    316.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/docker/tmp
overlay                   3.9G    316.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/docker/rootfs
tmpfs                     3.9G      4.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/acpid/tmp
overlay                   3.9G      4.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/acpid/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/binfmt/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/binfmt/rootfs
tmpfs                     3.9G      8.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dhcpcd/tmp
overlay                   3.9G      8.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dhcpcd/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/diagnose/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/diagnose/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dns-forwarder/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dns-forwarder/rootfs
tmpfs                     3.9G    316.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/docker.tar/70918e63c378be8683de4fc10c553798322a67941c518075202eab14ca0a8c55/containers/services/docker/tmp
overlay                   3.9G    316.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/docker.tar/70918e63c378be8683de4fc10c553798322a67941c518075202eab14ca0a8c55/containers/services/docker/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/http-proxy/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/http-proxy/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/kmsg/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/kmsg/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/procd/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/procd/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/sntpc/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/sntpc/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/socks/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/socks/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/trim-after-delete/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/trim-after-delete/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/volume-contents/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/volume-contents/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/vpnkit-forwarder/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/vpnkit-forwarder/rootfs
overlay                 149.4G     34.8G    106.9G  25% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged
overlay                 149.4G     34.8G    106.9G  25% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged
tmpfs                    64.0M         0     64.0M   0% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged/dev
shm                      64.0M         0     64.0M   0% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged/dev/shm
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged/sys/fs/cgroup
tmpfs                     3.9G         0      3.9G   0% /host/var/log
tmpfs                   796.0M    480.0K    795.5M   0% /host/var/run/linuxkit-containerd/containerd.sock
tmpfs                   796.0M    480.0K    795.5M   0% /host/var/run/linuxkit-external-logging.sock
grpcfuse                931.5G     14.3G    768.1G   2% /host/host_mnt
/dev/vda1               149.4G     34.8G    106.9G  25% /etc/resolv.conf
/dev/vda1               149.4G     34.8G    106.9G  25% /etc/hostname
/dev/vda1               149.4G     34.8G    106.9G  25% /etc/hosts
tmpfs                     3.9G         0      3.9G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                     3.9G         0      3.9G   0% /sys/firmware

I've never seen this happen before, it just started a couple of weeks ago. It doesn't happen all the time, everything works fine for hours, sometimes days, and then it crashes. None of my colleagues working on the same project has had the same problem.

Docker Desktop for Mac: v3.6.0 (67351)
Docker Engine: v20.10.8
Compose: v1.29.2
MacOS: v11.5.1 (20G80)

Anyone else experiencing this issue? Any suggestion on how I can further investigate the problem?

Is this a mounted volume? and could you please share your `docker-compose.yml` and `Dockerfile`? If it is a volume bind related problem, these may also be helpful: [1](https://stackoverflow.com/a/45688516/5492826), [2](https://stackoverflow.com/a/32785014/5492826) — emrekgn, Aug 19 '21 at 14:19
They are mounted volumes. With NGINX, it's just the official Docker image with a single file shared as a mounted volume (nginx.conf). It works for days without problems, the same configuration works for my colleagues, but in my case, every now and then, everything stops to work and it's because of I/O errors. — Alessandro Cappello, Aug 19 '21 at 15:13
How much disk space is available _within the embedded Linux VM_? — BMitch, Aug 20 '21 at 09:30
How can I check it @BMitch? With a quick `df -h` inside the mongo container, it says 150GB total/137GB available. — Alessandro Cappello, Aug 20 '21 at 09:33
You can try `docker run --rm -v /:/host busybox df -h` but I'm not sure on the mac if that mount will work. Share the full output as an edit, rather that a summary of one of the lines. — BMitch, Aug 20 '21 at 12:41
In the course of 5 hours, it looks like you've used up 21G of the allocated 150G of storage in the VM. I expect after a few days, sometimes hours, of that, you'll exhaust your available disk space. — BMitch, Aug 21 '21 at 13:55
Sorry for the late reply: I was waiting to see if that was the case. It is not. We have multiple projects, and depending on which ones I start, the disk space would vary. It crashed again now, and it still has 105GB available. — Alessandro Cappello, Aug 24 '21 at 08:52

score 0 · Accepted Answer · answered Sep 01 '21 at 16:59

0

It seems that the issue disappeared when I deactivated the option called Use gRPC FUSE for file sharing. I don't know if this is related to a problem on my disk or my machine, but I'm glad it works now.

answered Sep 01 '21 at 16:59

Alessandro Cappello

498
8
24

Sudden I/O errors with docker-compose

1 Answers1