2

I am trying to run a sript in the background:

nohup script.py > out 2> err < /dev/null &

The script (Python 3.4) does at some point:

answer = input('? ')

(it has a menu running in one of the threads)

And the nohup call is crashing with:

EOFError: EOF when reading a line

Because of the /dev/null redirection of stdin I imagine. If I run it without stdin redirection:

nohup script.py > out 2> err &

It crashes with:

OSError: [Errno 9] Bad file descriptor

If I run it with:

script.py > out 2> err

It works, but blocks my terminal (it is in the foreground)

If I run it with:

script.py > out 2> err &

It runs in the background alright, but it gets stopped as soon as the input call is reached.

What I would like is:

  • be able to redirect stdout and stderr to the filesystem
  • be able to put the script in the background
  • be able to move if to the foreground and interact with the menu normally (so stdin must be enabled somehow). stdout and stderr would still be redirected to the filesystem, but stdin would behave normally.
  • the script must run fine in the background and in the foreground (of course, the menu is not working in the background, because stdin is "frozen")

Basically, what I would like is that when it is in the background, stdin is kind of "frozen", and whenever it comes to the foreground it works again normally.

Is this possible? The solution does not need to involve nohup

blueFast
  • 41,341
  • 63
  • 198
  • 344
  • What exactly should it do while stdin is frozen except stop and wait for stdin to become available? – user2313067 Sep 16 '15 at 05:13
  • @user2313067: What I expect is that the `input` call waits until `stdin` is "unfrozen", and then reads from it. I would like to avoid the the program to be stopped just because there is no active stdin, it should just wait until stdin is active to continue reading from it. To say it another way: a "frozen" `stdin` should behave exactly the same as an `stdin` without data coming in. I do not know if this is possible - if there are primitives (at the kernel level?) to support this, but this is what I looking for. Maybe it is not possible at all ... – blueFast Sep 16 '15 at 08:02
  • @metatoaster: yes, handling `EOF` would solve the problem, but that means modifying the script. I can do that because I have the source code (and it is python, so you nearly always have the source code), but this is not practical in the general case: maybe there is no source code, maybe there are 2 million input calls all around the code. I would like to find a non-invasive solution. – blueFast Sep 16 '15 at 08:06
  • Either the application utilize `stdin` correctly as per expectation or it is simply incorrect. This is literally the foundation of all unix binaries. Also you still have not define what "run fine in the background" means, however by definition so far it correctly stopped as it expects further input. Also, the expectation is that `stdin` cannot simply be an `EOF`, so either fix that or your problem cannot be solved. General cases do not mean anything in this context because a broken binary is still a broken binary. – metatoaster Sep 16 '15 at 11:42
  • @metatoaster: It is not broken: the program just sees EOF and crashes, which is ok. What I want is a mechanism to avoid the program to *see* EOF. I want to be able to unplug the running program from the terminal, without having an EOF being sent. I want the program to keep a functioning stdin which is not connected to anything for a while, until it is brought back to the terminal's foreground. Not "connected for a while" means here that no input data will be generated, not that the `stdin` has been terminated. – blueFast Sep 16 '15 at 13:33
  • @metatoaster: with this mechanism, no EOF would be sent, so the program would run normally, waiting for data for the `input` call. The data will only start arriving once the `stdin` is re-connected again to the terminal (by putting the program in the foreground), and the user types something. – blueFast Sep 16 '15 at 13:34
  • Then you run it as `script.py > out 2> err &`, where it "runs in the background alright" until the input call is reached, where user can simply enter `%` to bring the task into foreground to send commands (via stdin attached to the terminal that started it), then send `^Z` to suspend it and keep it running in the background it again with `%&`. Do you need an answer with an example program as a demo to how this actually works? – metatoaster Sep 16 '15 at 14:20

1 Answers1

1

What you want (and how input works and fails on EOF under python) with an interactive menu means that you cannot safely pass a file as stdin when invoking your program. This means your only option is to invoke this like so:

$ script.py > out 2> err &

As a demonstration, this is my script:

from time import sleep
import sys

c = 0
while True:
    sleep(0.001)
    c += 1
    if c % 1000 == 0:
        print(c, flush=True)
    if c % 2000 == 0:
        print(c, file=sys.stderr, flush=True)
    if c % 10000 == 0:
        answer = input('? ')
        print('The answer is %s' % answer, flush=True)

Essentially, every second it will write to stdout, every two seconds it will write to stderr, and lastly, every ten seconds it will wait for input. If I were to run this and wait a bit over a seconds (to allow disk flush), and chain this together, like so:

$ python script.py > out 2> err & sleep 2.5; cat out err
[1] 32123
1000
2000
2000
$ 

Wait at least 10 seconds and try cat out err again:

$ cat out err
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
? 2000
4000
6000
8000
10000

[1]+  Stopped                 python script.py > out 2> err
$ 

Note that the prompt generated by input is also written to stdout, and the program effectively continued running up to where it is expecting stdin to give it data. You simply have to bring the process back into foreground by %, and start feeding it the required data, then suspend with ^Z (CtrlZ) and keep it running again in the background with %&. Example:

$ %
python script.py > out 2> err
Test input
^Z
[1]+  Stopped                 python script.py > out 2> err
$ %&
[1]+ python script.py > out 2> err &
$

Now cat out again, after waiting another ten seconds:

$ cat out
1000
...
10000
? The answer is Test input
11000
...
20000
? 
[1]+  Stopped                 python script.py > out 2> err
$

This is essentially a basic crash course in how standard processes typically functions in both foreground and background, and things just simply work as intended if the code handles the standard IO correctly.

Lastly, you can't really have it both ways. If the application expects stdin and none is provided, then the clear option is failure. If one is provided however but application got sent to background and kept running, it will be Stopped as it expects further input. If this stopped behaviour is unwanted, the application is at fault, there is nothing can be done but to change the application to not result in an error when EOF is encountered when executed with /dev/null as its stdin. If you want to keep stdin as is, with the application being able to somehow keep running when it is in the background you cannot use the input function as it will block when stdin is empty (resulting in process being stopped).


Now that you have clarified via the comment below that your "interactive prompt" is running in a thread, and since usage of input directly reads from stdin and you seem unwilling to modify your program (you asking for general case) but expects a utility to do this for you, the simple solution is to execute this within a tmux or screen session as they fully implement a pseudo tty that is independent from whichever console that started (so you can disconnect and send the session to the background, or start other virtual sessions, see manual pages) which will provide stdio that the program expects.

Finally, if you actually want your application to support this natively, you cannot simply use input as is, but you should check whether input can safely be called (i.e. perhaps making use of select), or by checking whether the process is currently in the foreground or background (An example you could start working from is How to detect if python script is being run as a background process, although you might want to check using sys.stdin, maybe.) to determine if input can be safely called (however if user suspends the task as input comes it would still hang as input waits), or use unix sockets for communication.

Community
  • 1
  • 1
metatoaster
  • 17,419
  • 5
  • 55
  • 66
  • Thanks for the effort, but this is not exactly what I need. Basically, my interactive menu is running in its own thread. I want that thread to wait for input *indefinitely* whenever the process is in the background, without crashing, and when it is brought to the foreground it should continue accepting input normally from stdin. But, when it is in the background (and thus, no stdin is available), it should not crash with "EOFError: EOF when reading a line". It should just wait for the input to come, as if the user were not yet inputing anything. – blueFast Sep 16 '15 at 17:22
  • The other threads, which are not interactive, should keep running normally, which is not the case right now because the interactive thread crashes the whole process. – blueFast Sep 16 '15 at 17:22
  • For your sample program, it would suffice if the `input` call would block the program, but *not* by stopping the process, but just by waiting for input, as any `input` call does ... **even whenever there is no stdin attached**. The `input` call should continue reading from `stdin` when it is re-attached (by bringing the process to the foreground) – blueFast Sep 16 '15 at 17:24
  • Ideally there would be a wrapper utility (let's call it `savestdin`), similar in concept to what `nohup` does, but with another purpose, namely keeping an `stdin` alive even for processes in the background, which can be re-attached to the terminal on demand. I would call my program like this then: `savestdin python script.py > out 2> err &`, and whenever the job is put to the foreground again, the terminal would be re-connected to the process `stdin`. I am not aware of any such utiliy. – blueFast Sep 16 '15 at 17:31
  • If you had mentioned the usage of thread in your question it would have been a lot more clear on what your actual problem is. Anyway, what you want is simply not possible with how your application as written as I had explained, you can't expect `input` to work with `stdin` "disconnected" (it will either see EOF and raise exception, or hang all threads indefinitely as you saw). Try spawning background processes instead of threads. – metatoaster Sep 16 '15 at 22:38
  • Alternatively, if you are looking for a wrapper utility, run your utility using [`tmux`](https://tmux.github.io/) or [`screen`](https://www.gnu.org/software/screen/), as it provides a fully implemented virtual tty that can be disconnected/reconnected via internal hotkeys (i.e. stdio is completely captured in its sessions). Which is why I said you can't just simply use `stdin` to do what you wanted to do, and you were not clear in your original question with **everything** you have (especially lack of code). – metatoaster Sep 16 '15 at 22:44
  • In my original question I do mention threads, as I know that is an important aspect of what I expect to achieve: "(it has a menu running in one of the threads)". Regarding lack of code: the only relevant code for this question is "input('? ')". If that can be kept running even "in the background", until it is put "back to the foreground" that is fine enough. There is no need to test with threads at all. – blueFast Sep 17 '15 at 07:02
  • And yes, tmux/screen do something similar to what I want, but are too heavy weight. I expected to find a small utility which just suspends stdin, as tmux/screen are able to do (which, by the way, prove that what I am asking for *is not impossible*) – blueFast Sep 17 '15 at 07:04
  • By not possible I mean within the context of your application (you can't just suspend and expect a standard read of stdin to not block the entire process) - you need to run it within a (pseudo) terminal and as far as I know tmux and screen are the only detatchable psuedo terminals I know of (also, a recommendation of software/tools is typically out of the scope of typical stackoverflow questions). Basically, write your own tmux or screen without the excess "heavy weight" package, or fix your application to not use [blocking I/O](http://stackoverflow.com/a/717586/2904896) (i.e. no `input`). – metatoaster Sep 17 '15 at 07:58
  • Also, my complaint about lack of mention of threads: I mean mentioning the suspension of the process plus all the background threads and that you want those threads to keep running (i.e. keep the process alive somehow, but this was all covered up by the length of the question (thread was mentioned once) and muddled by your misunderstanding of how stdio actually works and my vain attempt in trying to explaining this to you, so I apologise). – metatoaster Sep 17 '15 at 08:04
  • @jeckyll2hide Someone else asked a similar question and got me to think about this again. Turns out it's a lot easier than I thought - please check out my [other answer](http://stackoverflow.com/a/32899461/2904896) which contains a short yet complete code example to demonstrate the solution. – metatoaster Oct 02 '15 at 03:35