12

Python has its own RotatingFileHandler which is supposed to automatically rotate log files. As part of a linux application which would need to rotate it's log file every couple of weeks/months, I am wondering if it is any different than having a config file in logrotate.d and using a WatchedFileHandler instead.

Is there any difference in how they operate? Is one method safer, more efficient, or considered superior to the other?

Ciro Santilli OurBigBook.com
  • 347,512
  • 102
  • 1,199
  • 985
devhallo
  • 337
  • 3
  • 13
  • `logrotate.d` depends on the logger closing and reopening its file pointer on each log event. I'm not sure if the standard python file logger does that. – tdelaney Mar 17 '15 at 16:52
  • @tdelaney it doesn't, but there's a good alternative inside `logging`, see the bottom of my answer – loopbackbee Mar 17 '15 at 17:10
  • I noticed you edited you question to mention `WatchedFileHandler`. Did you mean `RotatingFileHandler`? – loopbackbee Mar 19 '15 at 13:21
  • 1
    no. `WatchedFileHandler` is used with `logrotate`. `RotatingFileHandler` takes care of rotating the logs itself, and replaces `logrotate`. – devhallo Mar 19 '15 at 14:38

2 Answers2

9

What is the intended audience of your program?

If you're creating a desktop application and most users can't be expected to read the logs, you should handle it for them. Not only rotating, but also deleting old ones - you don't want to fill the poor user's hard drive!

On the other hand, if the audience is experienced UNIX sysadmins, you'll have to take a different approach.

Sysadmins will need features you cannot possibly anticipate. Send them by email, write them to append-only storage, you name it. For this audience, it's best if your logging is as flexible as possible. Flexible (in UNIX) means simple - so just write to a file and consider it done.

Also, sysadmins don't want to re-learn how to do logging all over again. Even if you want to provide this kind of feature, make sure the default is reasonable within this assumption.

Finally. tdelaney raised a important point: the standard FileHandler doesn't pay much attention to the file it's writing to. You should use a WatchedFileHandler, which was written specifically for this purpose

loopbackbee
  • 21,962
  • 10
  • 62
  • 97
  • That's a good argument for the `syslog` and `NTEventLog` handlers which log to the expected places. The app shouldn't have write access to `\var\log` but can write its logs there anyway via the standard syslog calls. – tdelaney Mar 17 '15 at 17:15
  • Good point, it doesn't directly my question though. But I am guessing you mean advanced *nix users would expect a log rotating mechanism to be handled by logrotate, if at all? – devhallo Mar 17 '15 at 17:51
  • @tdelaney `syslog` and `NTEventLog` are indeed a better choice if you want to explicitly write code for different platforms – loopbackbee Mar 17 '15 at 18:21
  • @devhallo yes, that's my point. It's not unheard of having a daemon perform rotation, but it's inconvenient. Which aspects of your question do you feel are not addressed here? – loopbackbee Mar 17 '15 at 18:23
  • 1
    @goncalopp your answer only outlines the way a programmer should think about logging, depending the targeted audience. It doesn't answer the question of wether to use python's `RotatingFileHandler` or the `logrotate` tool, as both could be used to accomplish the same task. I have accepted it though, since your comment makes things a bit clearer, but you might want to edit it to directly answer the question for future reference. – devhallo Mar 18 '15 at 22:00
  • It's not entirely unlikely that you may want an action taken when a log is rotated (e.g. restarting `rsyslog`). I'm unsure if this is possible with the python logging framework; it is a core functionality of `logrotate`. – Ben Mar 02 '17 at 14:48
4

RotatingFileHandler allows a log file to grow up to size N, and then immediately and automatically rotates to a new file.

logrotate.d runs once per day usually. If you want to limit a log file's size, logrotate.d is not the most helpful because it only runs periodically.

mbrandeis
  • 250
  • 1
  • 2
  • You should note that logrotate does have a size option, which will rotate a log once it reaches/exceeds a set size.. Reference site: http://www.thegeekstuff.com/2010/07/logrotate-examples/ – Angry 84 Jan 06 '16 at 23:14
  • But logrotate will still only perform the rotation when it runs during its set time. So there is a lag period where the log can continue to grow. Changing logrotate to run hourly will mitigate this somewhat of course. – pfctdayelise Mar 06 '16 at 23:23