Logging and rotating stdout and stderr in Python

Question

I am a Linux newbie and I'm running the following Linux command:

nohup python my_script.py > my.out 2> my.err

and everything works as expected; all the output from my python (2.7) print statements get logged in the file called my.out.

The only problem is that after several days of running the file, my.out, gets very large. Is there a way (at the Linux shell level) I can have a new "my.out" created, say once a day, with a time stamp in the file name?

You could also rewrite your script to use Python's logging module, with the [RotatingFileHandler](https://docs.python.org/2/library/logging.handlers.html#rotatingfilehandler) — FamousJameous, Jun 29 '17 at 14:00
I strongly recommend @FamousJameous suggestion however if you want to do it on the OS level you can create a [cron job](https://help.ubuntu.com/community/CronHowto). — a_guest, Jun 29 '17 at 14:06
Also see [Logger configuration to log to file and print to stdout](https://stackoverflow.com/q/13733552/608639) and [How to redirect stdout and stderr to logger in Python](https://stackoverflow.com/q/19425736/608639), — jww, Jun 29 '17 at 20:50

score 2 · Accepted Answer · answered Jun 29 '17 at 14:24

2

You should use logging module in your script :

import logging.handlers

LOG_FILE_NAME = '/var/log/my_script.log'
LOGGING_LEVEL = logging.INFO

formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)s %(message)s')
handler = logging.handlers.RotatingFileHandler(LOG_FILE_NAME, mode='a', maxBytes=5000000, backupCount=5)
handler.setFormatter(formatter)
log = logging.getLogger("my_script")
log.addHandler(handler)
log.setLevel(LOGGING_LEVEL)

In this example, we configure a log file rotation of 5MB with a retention of 5 files. So each time your log file reach 5MB, the script creates a new log file and archive the old one (5 files max).

Here is an example of code to log data in your log file :

def do_something():
    # do stuff here
    log.info("Something has been done")

Once you've done that, simply run your python script :

python my_script.py

Everything will be logged in the file you want with automatic log rotation.

If you prefer a rotation based on time, please refer to TimedRotatingFileHandler

https://docs.python.org/2/library/logging.handlers.html#timedrotatingfilehandler

answered Jun 29 '17 at 14:24

Nab Ilovich

360
5
14

RotatingFileHandler worked very nicely. I have one question, is there a way to specify an unlimited backupCount? Thanks also to FamousJameous – Phil O Jun 29 '17 at 16:09
@PhilO According to the documentation it's not possible. You can still backup your log files so they are not erased during log rotation. – Nab Ilovich Jun 29 '17 at 16:26
Is there a way to log to the file and the console? The latter would help with debugging. – Phil O Jun 29 '17 at 16:34
`log` will save data in your log file only. You can still redirect the std output to a file, but i dont see the point. If you make a good use of the command `log` you won't need to redirect console. And to debug, use a debuger :) – Nab Ilovich Jun 29 '17 at 16:40
The log line is in a loop? If not you can ask another question for that maybe. – Nab Ilovich Jun 29 '17 at 16:56
Stupid programmer error I was instantiating the logger 3 times! – Phil O Jun 29 '17 at 16:57
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/147967/discussion-between-phil-o-and-nab-ilovich). – Phil O Jun 29 '17 at 17:13
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/148052/discussion-between-phil-o-and-nab-ilovich). – Phil O Jun 30 '17 at 11:50

score 2 · Answer 2 · answered Jun 29 '17 at 14:36

What you want is what Apache fellows did with rotatelogs. Extract:

rotatelogs is a simple program for use in conjunction with Apache's piped logfile feature. It supports rotation based on a time interval or maximum size of the log.

Synopsis

rotatelogs [ -l ] [ -L linkname ] [ -p program ] [ -f ] [ -t ] [ -v ] [ -e ] [ -c ] [ -n number-of-files ] logfile rotationtime|filesize(B|K|M|G) [ offset ]
...
rotationtime The time between log file rotations in seconds. The rotation occurs at the beginning of this interval. For example, if the rotation time is 3600, the log file will be rotated at the beginning of every hour; if the rotation time is 86400, the log file will be rotated every night at midnight.

For your use case, you could do:

nohup python my_script.py 2> my.err | rotatelogs -l my.out.%Y.%m.%d 86400

to have one file per day changing file at 0h00 local time.

You can find it in the Apache HTTPD project which exists for almost any common architecture

Marc Sànchez · Answer 3 · 2017-06-29T14:23:26.560

1

Yes it is, use command date to add a date inside the name of file, like :

nohup python my_script.py > my_`date +"%d_%m_%y"`.out 2> my.err

4 example I've done the next example with the following command :

echo "hello" > name_`date +"%m_%y"`

and the name of the file is name_06_17

You can see more information about date command typing :

man date

This will allow you to create different files, one per month.

hope will help!

edited Jun 29 '17 at 14:23

answered Jun 29 '17 at 14:17

Marc Sànchez

11
3

Yes, but I'd like the file to be renamed each day automatically. – Phil O Jun 29 '17 at 14:19
if you do the same command with the day in front will do this for you. You have to use doble redirection ">>" to add info to an existing file. – Marc Sànchez Jun 29 '17 at 14:21
nohup python my_script.py > my_`date +"%d_%m_%y"`.out 2> my.err – Marc Sànchez Jun 29 '17 at 14:22
2

@MarcSànchezThis won't create a separate file for each day, but would require to rerun the command every day. – a_guest Jun 29 '17 at 14:24
Yup, I don't want to do that. – Phil O Jun 29 '17 at 14:49
shure, better using a log file. – Marc Sànchez Jun 29 '17 at 16:19

Logging and rotating stdout and stderr in Python

3 Answers3