0

I need to only allow the file to contain 5000 lines of data before deleting the oldest data as the python script outputs new data (Similar to the 'tail command in bash'). I would like to be able to observe the data as it's being generated by viewing the text file that the script is outputting to. I need it to be able to run in the background without the need for the terminal, or in this case ssh session, to be active. At the beginning of the python script that outputs all the data, I tried something along the lines of: 

import sys
sys.stdout = open(tail -c 1G > 'output.txt', 'w', buffering=1)

But to no avail. I am not opposed to having a bash script handle the python file, but I would prefer to simply run the python script and have it's output directed to a separate file as mentioned above if possible.

Jon
  • 19
  • 4
  • 1
    Are you looking for Rotating file handler https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler – Luv May 08 '20 at 06:27
  • with standard `open()` you would have to manually count lines which you write to file and rewrite all to new file when you get more then 5000 lines - to skip beginning lines. – furas May 08 '20 at 06:57
  • Maybe. I just read your link. Is there a way to have the entire python script output to the log file? For instance, have every print statement output to the log file. Also, is the log file updated as the python file outputs data? – Jon May 08 '20 at 06:57

1 Answers1

0

You're describing a circular buffer, which is not a data structure that can be backed by a file. Files can be written to (overwriting some existing content) or appended to, but it's not possible to shift bytes earlier in the file. Commands that look like they do this typically write to a temporary file and then copy the contents back to the original file.

As @Luv suggested, the typical approach to "persist the last n lines of data to disk" is to rotate between several files. You can do that in Python (their suggestion of using RotatingFileHandler seems reasonable) or in Bash, such as with rotatelogs.

dimo414
  • 47,227
  • 18
  • 148
  • 244