0

I want to write a shell script that executes a python script and redirects its output with a timestamp to a different log file every day. The python script should run forever without stopping, it prints small text to the terminal every 10 seconds.

Here is how far I have come:

filename="log-$(date +%Y-%m-%d).log"
python3 -u HelloWord_every10secs.py 2>&1 | 
while IFS= read -r line; 
    do echo "$(date +%x__%H:%M:%S:%3N) $line"; 
    filename="log-$(date +%Y-%m-%d).log"
    done >> "log/$filename" #u unbuffered  output, for logging, add time stuff

This small code redirects the output of the python code to a log file in log/ directory and ads timestamps. The only issue that is missing it cannot rotate the log file over each day with a different date ending.
I except to have in log/ directory:

log-2022-11-20.log
log-2022-11-21.log
log-2022-11-22.log
... etc.

But I have only one with the date when I run the script.

I know that there is a Ubuntu built-in solution called lograte for these things, but I would like to solve this problem without that.

NNN123
  • 1
  • 2

1 Answers1

0

Rather than controlling log files externally, I would recommend using TimedRotatingFileHandler instead.

Example:

import logging
from logging.handlers import TimedRotatingFileHandler

log = logging.getLogger(__name__)

def init_file_handler(log_path, logger=None):
    handler = TimedRotatingFileHandler(log_path, when='midnight')
    handler.setFormatter(logging.Formatter('%(asctime)s %(message)s'))
    if logger is None:
        logger = logging.getLogger()  # This will be the root logger, which will capture all logs
    logger.setLevel(logging.INFO)  # Write INFO and above; use DEBUG or NOTSET for more
    logger.addHandler(handler)

Using the above, if we run this:

>>> init_file_handler('test.log')

>>> log.info('foo')

The result will be:

$ cat test.log
2022-11-20 07:43:32,785 foo

The first log line written after midnight will trigger the roll-over, where test.log from this example would be renamed to test.log.2022-11-20, and a new test.log would be created.

It is possible to change the time format in the logged events (and the overall format of each line), and it is also possible to adjust the date format for old log files.

Additional resources:

dskrypa
  • 1,004
  • 2
  • 5
  • 17
  • Thanks for a suggestion, I put it as backup solution if controlling them exeternally is not possible. – NNN123 Nov 21 '22 at 17:29
  • Counter point: why have 2 moving parts instead of 1? If you want the non-timestamped content on stdout/stderr, you can configure a different formatter for stdout/err compared to the one used for the file. It's generally easier to maintain something like this if you don't need to maintain a separate wrapper script in a different programming language. – dskrypa Nov 21 '22 at 19:18