0

I have a Python script running on a server through SSH with the following command:

nohup python3 python_script.py >> output.txt

It was running for a very long time and (probably) created useful output, so I want to force it to stop now. How do I make it write the output it has so far, into the output.txt file?

The file was automatically created when I started running the script, but the size is zero (nothing has been written in it so far).

Samuel
  • 43
  • 7
  • If you're using python logging to output text, then it's probably outputting to STDERR. You're only logging STDOUT. You should run it like this to capture both: `nohup python3 python_script.py >> output.txt 2>&1 &` – Robert Seaman May 29 '18 at 19:03
  • Thank you for your help. Now it finished by it self, and wrote everything nicely into the file. I just don't understand why it didn't occasionally write the output to the file while it was running – Samuel May 29 '18 at 20:01

1 Answers1

2

as Robert said in his comment, check that the output you are expecting to go to the file is actually making it there and not stderr. If the process is already running and has been for a long time without any response or writes into your output file, I think there are 3 options:

  1. It is generating outout but it's not going where you are expecting (Roberts response)
  2. It is generating output but it's buffered and for some reason has yet to be flushed
  3. It's hasn't generated any output

Option 3 is easy: wait longer. Options 1 & 2 are a little bit tricky. If you are expecting a significant amount of output from this process, you could check the memory consumption of the python instance running your script and see if it's growing or very large. Also you could use lsof to see if it has the file open and to get some idea what it's doing with it.

If you find that your output appears to be going somewhere like /dev/null, take a look at this answer on redirecting output for an existing process.

In the unlikely event that you have a huge buffer that hasn't been flushed, you could try using ps to get the PID and then use kill -STOP [PID] to pause the process and see where you can get using GDB.

Unless it would be extremely painful, I would probably just restart the whole thing, but add periodic flushing to the script, and maybe some extra status reporting so you can tell what is going on. It might help too if you could post your code, since there may be other options available in your situation depending on how the program is written.

Z4-tier
  • 7,287
  • 3
  • 26
  • 42
  • Thank you for your help! It is the 2nd option definitely. The code doesn't refer to any files directly. The only time I refer to a file with ">> output.txt". So you can imagine the script as a giant for-loop that prints something to std out every few minutes. I am not exactly sure how to flush the file in this case? I know how to do it only when I referred to the file within the script by using sth like f = fopen(...) – Samuel May 29 '18 at 20:00
  • Linux will write any STDOUT to the file promptly, but python could be holding onto it in the buffer before it reaches STDOUT. @SamuelBosch It would help if you told us how you're logging/printing from within your python program. You should have a look at the `logging` module, and see if you can use that to either log to STDERR, or directly to a file. – Robert Seaman May 29 '18 at 20:05
  • For simplicity, just assume that the program looks like this: for i in range(large_number): print(large_number) do_some_time_consuming_task() When I run this program, it would output 0,1,2,3,... (in separate lines), but when I run this as above, it waits until the program is done and then writes the whole output to output.txt – Samuel May 29 '18 at 22:06
  • @RobertSeaman You are right, that's a very likely situation. If python is using line buffering and you replace that `print()` with a `write()`, you lose the newline that print gives for free. A 4k buffer will hold a decent number of ints before it ever hands them off to the OS. @SamuelBosch If you _really_ want to get the contents of that buffer, you'll either need to wait until it's full, or (if you're on linux) you could do a `ulimit -C unlimited` and then a `kill -3 ` to dump a core file and then try and get your data out of that...it's a lot of work, depends how bad you want it :) – Z4-tier May 30 '18 at 02:35