0

My microservice status is down, and I login to docker container:

docker exec -it {container_id} /bin/bash

The app in container logs output:

2019-09-04 14:51:02.244  INFO 61 --- [DiscoveryClient-CacheRefreshExecutor-0] com.netflix.discovery.DiscoveryClient    : Force full registry fetch : false
2019-09-04 14:51:02.244  INFO 61 --- [DiscoveryClient-CacheRefreshExecutor-0] com.netflix.discovery.DiscoveryClient    : Application is null : false
2019-09-04 14:51:02.244  INFO 61 --- [DiscoveryClient-CacheRefreshExecutor-0] com.netflix.discovery.DiscoveryClient    : Registered Applications size is zero : true
2019-09-04 14:51:02.244  INFO 61 --- [DiscoveryClient-CacheRefreshExecutor-0] com.netflix.discovery.DiscoveryClient    : Application version is -1: true
2019-09-04 14:51:02.244  INFO 61 --- [DiscoveryClient-CacheRefreshExecutor-0] com.netflix.discovery.DiscoveryClient    : Getting all instance registry info from the eureka server
2019-09-04 14:51:02.326  INFO 61 --- [DiscoveryClient-HeartbeatExecutor-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_APOLLO-ADMINSERVICE/b24984e8c669:apollo-adminservice:8090 - Re-registering apps/APOLLO-ADMINSERVICE
2019-09-04 14:51:02.326  INFO 61 --- [DiscoveryClient-HeartbeatExecutor-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_APOLLO-ADMINSERVICE/b24984e8c669:apollo-adminservice:8090: registering service...
2019-09-04 14:51:02.470  INFO 61 --- [DiscoveryClient-HeartbeatExecutor-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_APOLLO-ADMINSERVICE/b24984e8c669:apollo-adminservice:8090 - registration status: 204
2019-09-04 14:51:02.485  INFO 61 --- [DiscoveryClient-CacheRefreshExecutor-0] com.netflix.discovery.DiscoveryClient    : The response status is 200
2019-09-04 14:51:02.747  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)
2019-09-04 14:51:32.068  INFO 61 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
2019-09-04 14:51:32.802  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)
2019-09-04 14:52:02.857  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)
2019-09-04 14:52:32.069  INFO 61 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
2019-09-04 14:52:32.912  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)
2019-09-04 14:53:02.968  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)
2019-09-04 14:53:32.070  INFO 61 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
2019-09-04 14:53:33.026  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)
2019-09-04 14:54:03.083  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)
2019-09-04 14:54:32.071  INFO 61 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
2019-09-04 14:54:33.149  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)
2019-09-04 14:55:03.247  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)
2019-09-04 14:55:32.072  INFO 61 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
2019-09-04 14:55:33.306  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)
2019-09-04 14:56:03.360  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)
2019-09-04 14:56:32.074  INFO 61 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
2019-09-04 14:56:33.417  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)
tail: unrecognized file system type 0x794c7630 for ‘apollo-adminservice.log’. please report this to bug-coreutils@gnu.org. reverting to polling
2019-09-04 14:57:03.558  WARN 61 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 0 bytes (threshold: 10485760 bytes)

and I checkout the container disk:

root@b24984e8c669:/opt/logs/100003172# df -lh
Filesystem      Size  Used Avail Use% Mounted on
overlay          40G   38G     0 100% /
tmpfs            64M     0   64M   0% /dev
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/vda1        40G   38G     0 100% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           7.8G     0  7.8G   0% /proc/acpi
tmpfs           7.8G     0  7.8G   0% /proc/scsi
tmpfs           7.8G     0  7.8G   0% /sys/firmware

what should I do do fix the problem? how to tweak the container disk usage?thanks!

Dolphin
  • 29,069
  • 61
  • 260
  • 539

1 Answers1

1

how to tweak the container disk usage?

In fact, this not related to container disk usage, this related to your docker host's disk usage.

If use docker info | grep Dir on host to have a look, you could see something like next:

Docker Root Dir: /var/lib/docker

This folder is the overlay filesystem docker used for all your containers.

So, overlay 40G 38G 0 100% / just means the host disk which for /var/lib/docker is full. Your 40G partition is not enough, what you could do is next:

Option1:

If you have other partition which is bigger than 40G, change the storage location of docker, something like this.

Option2:

Just free some space (Remove unused big picture, documents, installers etc) on current disk (40G). Additional, I guess you may have a lots of unused images & containers, then you may could use docker system prune -a to clean them. But pay attention to next:

WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all images without at least one container associated to them
- all build cache

atline
  • 28,355
  • 16
  • 77
  • 113