1

I've got a Docker container currently running in production on a CentOS 7 VM. We have encountered a problem where the logs of the container are filling up the host drive (the log files found at /var/lib/docker/{continer_name}) over time and causing the container to become unresponsive forcing us to clear logs on the host in order to enable it to continue processing.

We can't take the container down, meaning I can't just bring it back up using the --log-opt flag to set up some log rotation options.

We've tried using logrotate, but the nature of the container means the logs are being written to regularly and what we find is often the logs are rotated, but the original file does not decrease in size due to being written to whilst the rotation is underway.

I'm trying to find a solution to this problem where we can set up some kind of task that will clear the logs down to a specific file size. Any help is greatly appreciated.

Maltanis
  • 513
  • 1
  • 5
  • 13
  • Possible duplicate of [How to clean Docker container logs?](https://stackoverflow.com/questions/41091634/how-to-clean-docker-container-logs) – atline Sep 05 '18 at 08:18
  • I'm going to say not a duplicate as those solutions all involve taking the container down, or using settings that require starting the container up again. I literally cannot remove this container. I can stop and start it currently, but once live we need a way to actually clear the logs without bringing the container down. – Maltanis Sep 05 '18 at 08:25
  • How about `configure docker to automatically rotate logs with the following in an /etc/docker/daemon.json file`, the answer by BMitch? – atline Sep 05 '18 at 08:27
  • @atline from my understanding that only works for new containers? I have an exisiting container, so those changes won't be made unless I start from scratch, which as stated I can't do. – Maltanis Sep 05 '18 at 08:31
  • "We can't take the container down" one of the key designs of containers is to be ephemeral. Sounds like you've created a pet and would be better served by redesigning the HA of your app. – BMitch Sep 05 '18 at 08:32
  • Yes, seems for new container, if do not want to redesign, this solution really cannot meet your requirements. But @BMitch 's solution is good for new design really. – atline Sep 05 '18 at 08:35
  • @BMitch it's not my decision to say we can't take it down. The deployment team are adament they cannot just take the container down and start up again with the new options without creating a mountain of work for themselves, despite my best efforts to create an easy to deploy product. Thus I'm forced into this situation of finding a solution for a problem that shouldn't exist in the first place! – Maltanis Sep 05 '18 at 08:38

1 Answers1

0

i would suggest mounting the containers logs directory to a host directory, and there you can schedule whatever task to zip/move/delete the log files...

David Gidony
  • 1,243
  • 3
  • 16
  • 31
  • 1
    I realised my question didn't explain well enough that these logs are on the host machine, mounted on a seperate volume. These would be found in /var/lib/docker/{continer_id} – Maltanis Sep 05 '18 at 08:30
  • i assume the containers application is responsible for the logs being written, might find a setting on the app level to limit the logs, or at least set maximum log file size , and make the log file split after x mb's – David Gidony Sep 05 '18 at 08:33