0

Having an issue with one of our Gitlab Servers running in Docker. Ran a build today that broke because of no space left on device. Come to find out that /var is out of storage.

A du -h --threshold=1G in /var will quickly show that the space is being used by overlay2 in lib/docker/overlay. All of the prune commands one may find (docker system prune, docker volume prune, etc) don't leave a dent in the storage being consumed. So what gives? Any data in the Gitlab server is being persisted elsewhere, but something docker related is gradually filling up space on the filesystem, I just don't know what it is. Can someone help me understand what is going on?

Thank you!

NickHayMan
  • 1

1 Answers1

0

You can do many things to overcome this issue:

  1. Increase the disk capacity.
  2. Clearing docker logs for gitlab related services. Be-careful about your company log retention period. Detailed guide in this SO thread
Sachith Muhandiram
  • 2,819
  • 10
  • 45
  • 94
  • 1
    Increasing the disk capacity doesn’t solve the underlying problem however. Even if you increase it to a TB it will eventually fill up. The Log directory for the container didn’t fill the space. – NickHayMan Feb 02 '23 at 05:00
  • @NickHayMan In this case, its gitlab repositories taking space. You can not do anything about those. Gitlab also generates backups, check if those are also configured to `/var` dir etc. – Sachith Muhandiram Feb 02 '23 at 05:40