6

I have a question, and I could'nt find help anywhere on stackoverflow or the web.

I have a program (celery distributed task queue) and I have multiple instances (workers) each having a logfile (celery_worker1.log, celery_worker2.log).

The important errors are stored to a database, but I like to tail these logs from time to time when running new operations to make sure everything is ok (the loglevel is lower).

My problem: these logs are taking a lot of disk space. What I would like to do: be able to "watch" the logs (tail -f) only when I need it, without them taking a lot of space.

My ideas until now:

  • outputing logs to stdout, not to a file: not possible here since I have many workers outputing to different files, but I want to tail them all at once (tail -f celery_worker*.log)
  • using logrotate: it is an "OK" solution for me. I don't want this to be a daily task but would rather not put a minute crontab for this, and more, the server is not mine so that would mean some work on the admin-sys side
  • using named pipes: it looked good at first sight but I didn't know that named pipes (linux FIFO) where blocking. Hence, when I don't tail -f ALL of the pipes at the same time, or when I just quit my tail, the writing operations from the logger are blocked.

Is there a way to have a non-blocking named pipe, which would just throw to stdout when tailed, and throw to /dev/null when not?

Or are there technical difficulties to such a type of pipe? If there are, what are they?

Thank you for your answers!

Noé Malzieu
  • 2,530
  • 3
  • 22
  • 27
  • possible duplicate of [Linux non-blocking fifo (on demand logging)](http://stackoverflow.com/questions/7360473/linux-non-blocking-fifo-on-demand-logging) – Piotr Dobrogost Jan 29 '15 at 21:26

2 Answers2

1

Have each worker log to stdout, but connect each stdout to a utility that automatically spools and rotates logs based on size or time. multilog and svlogd are examples of such. For those programs, you'd merely tail the "current" log file.

You're right that logrotate is not quite the right solution for the problem you have.

Named pipes won't work as you want. At best, your writers could fill up their pipes and then discard subsequent logs, which is the inverse of the behavior you want.

pilcrow
  • 56,591
  • 13
  • 94
  • 135
  • Thank you @pilcrow for your answer. All 5 workers are launched in background via a single command and I don't want to have a screen with tabs always open, so how can I have them log to different stdouts? And are you confirming that "non blocking named pipes redirecting to /dev/null when not read" do not exist as such ? Thanks – Noé Malzieu Feb 27 '13 at 22:38
  • @noe, regarding shared or separate stdout ... that depends. We don't know enough about their invocation to say. Regarding such named pipe semantics not existing, correct. – pilcrow Feb 28 '13 at 03:08
1

You could try shared memory device man:shm_overview or perhaps a number of them. You need to organise them as circular buffers so they'd store last N kb of your log and whenever you read them with reader it will output everything to your console. This approach is adopted by busybox's syslog/logread suit (see logread.c).

event
  • 379
  • 1
  • 4