0

I am using Python 3.8.9 and need to create a subprocess that I can send multiple commands to. My code is as follows:

sub1=subprocess.Popen('echo a\n', shell=True, stderr=subprocess.PIPE, stdin=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True)
sub1.stdout.readline()
sub1.stdout.flush()
sub1.stdin.flush()
sub1.stderr.flush()
sub1.stdin.write('echo b\n')
sub1.stdout.readline()

And this prints

'a\n'
7
''

while it should print

'a\n'
7
'b\n'

I don't understand why it seemingly isn't executing the second command, or quite frankly giving any output at all after running another command. I heard some people saying that .write is having issues in Python 3 but that .communicate does work, so I tried that but to no avail

sub1=subprocess.Popen('echo a\n', shell=True, stderr=subprocess.PIPE, stdin=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True)
sub1.stdout.readline()
sub1.stdout.flush()
sub1.stdin.flush()
sub1.stderr.flush()
out, err= sub1.communicate(input='echo b\n', timeout=15)
print(out)
print(err)

Both out and err printed

''

I couldn't find any extant question that covers this issue so I hope it is something that can be simply resolved.

Owen Burns
  • 120
  • 10
  • I can't reproduce the output you're showing using the posted code. – Woodford May 06 '21 at 17:10
  • Where is your second flush for 'b'? – Clarus May 06 '21 at 17:21
  • Why do you think that code should read a second command from stdin at all? The command `sh -c 'echo a'` (to explain: `['sh', '-c']` is added to the front of the argument list when you use `shell=True`) _only_ echos `a` and exits; it does not stick around and read more commands to run. – Charles Duffy May 06 '21 at 17:23
  • (also, `shell=True` uses `sh`, not `bash`, by default, so the bash tag should not be used here) – Charles Duffy May 06 '21 at 17:25
  • Instead of running `echo a` as your command (telling the shell to exit as soon as it echos `a`), switch to `shell=False` and make your command explicitly be something like `['bash', '-l', '-i']`, and feed _all_ the commands, including the first `echo a` one, on stdin. The linked duplicate discusses some of the problems you'll run into when doing this and how to work through them. – Charles Duffy May 06 '21 at 17:28
  • @Woodford What output are you getting? If what I'm trying to achieve is working on your machine and not working on mine I would also like to know what python version you're using – Owen Burns May 06 '21 at 17:29
  • @CharlesDuffy In reading the documentation I saw nothing about the subprocess closing after running the command in args, and every example doing what I'm trying to do used more or less the same code I'm using; having something in args is required so if the suprocess was going to close after that is run what would be the point of the communicate command existing? – Owen Burns May 06 '21 at 17:31
  • @Clarus is a second flush required before showing the output? In response to your comment I tried running sub1.stdin.flush() after the communicate command and after the write command, and for the former got this error: `ValueError: I/O operation on closed file` and for the latter `OSError: [Errno 22] Invalid argument` – Owen Burns May 06 '21 at 17:37
  • subprocess close in the same way you run a shell script in command line. it finishes and quits. Not a persistent thing. Instead of echo, use 'cat' – Bing Wang May 06 '21 at 17:38
  • @BingWang How do I keep the shell open for multiple commands? I have seen others do it with essentially the same code, and if there was no way to keep the shell open I don't see why a communicate command which can only be run after the initial args have been passed would exist. – Owen Burns May 06 '21 at 17:43
  • @Falcon007, why would it be in the documentation? That's the way shells _always_ work when you run `sh -c 'some command'` -- it runs `some command` and then exits; it's so normal that it would only need to be documented only if it were the _other_ way. This isn't just how they work in Python, it's how they work _everywhere_. Similarly, when you run `ssh somehost 'ls /tmp'`, it doesn't `ls /tmp` and then wait for more input, it does `ls /tmp` _and then exits_ and returns you to your original shell. – Charles Duffy May 06 '21 at 17:50
  • @Falcon007, ...I don't understand what you're saying about `communicate()` being pointless. The big advantage of `communicate()` is to be able to handle reading from both stdout and stderr without the order of the read operations in the parent process needing to match the order of write operations in the child to prevent deadlocks. – Charles Duffy May 06 '21 at 17:54
  • Huh? The value of `communicate()` is obvious; it feeds stdin to the child, reads its stdout and stderr (simultaneously not one-at-a-time, so it works no matter if the child writes stdout first, writes stderr first, or mixes the two), and returns them both. I don't understand how you're saying it's impossible to use. – Charles Duffy May 06 '21 at 17:55
  • @CharlesDuffy The communicate command, according to their documentation "Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached." What would be the point of a command that sends data to stdin if sending data to stdin is impossible after creating the subprocess? – Owen Burns May 06 '21 at 17:56
  • I never told you sending data to stdin was impossible after creating the subprocess. – Charles Duffy May 06 '21 at 17:56
  • What I did tell you is that the command `echo a` _doesn't try to read from stdin_. Obviously, you can't send data to stdin of a process that doesn't try to read from it; the process will just ignore that data. – Charles Duffy May 06 '21 at 17:57
  • To give you a concrete example, let's say you ran `subprocess.Popen('read line; echo "The line we just read is: <$line>"', stdin=PIPE, stdout=PIPE, stderr=PIPE)` -- there, `communicate('some input')` would be making `some input` be the value that goes into `$line`. – Charles Duffy May 06 '21 at 17:58
  • I know that echo a doesn't try to read from stdin. That's obvious because that's just how the command line works. I am trying to make the Popen object read its own stdin from the pipe created for it in the constructor, and from what you're telling me that isn't possible – Owen Burns May 06 '21 at 17:58
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/232055/discussion-between-charles-duffy-and-falcon-007). – Charles Duffy May 06 '21 at 17:59
  • But when passing the communicate command it should be communicate(input='some input', timeout=some number) which sends 'some input' to [popen object].stdin – Owen Burns May 06 '21 at 18:00
  • 1
    I am not sure what a use case of wrapping around a live shell. Did you hear pexpect? – Bing Wang May 06 '21 at 18:10
  • (to amplify @BingWang's suggestion -- pexpect is a really excellent library; I generally suggest _not_ wrapping a live shell when it can be avoided; but if for some reason you must, pexpect is great). – Charles Duffy May 06 '21 at 18:29
  • @BingWang It doesn't look like that package has strong windows support, unfortunately – Owen Burns May 07 '21 at 13:11
  • by echo I assume you are using sort of *Nix. If you need user interaction, you can run subprocess.Popen again and again to simulate pexpect. Below is, anyway, an example how you can wrap a live shell, I would use it only when other means are not feasible. – Bing Wang May 07 '21 at 13:57
  • ````import subprocess sub1=subprocess.Popen('bash', shell=True, stderr=subprocess.PIPE, stdin=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True) sub1.stdin.write('echo hello\n') sub1.stdin.flush() sub1.stdout.flush() a=sub1.stdout.readline() sub1.stdin.write('echo b\n') sub1.stdin.flush() b= sub1.stdout.readline() print(a,b) ```` – Bing Wang May 07 '21 at 13:58

0 Answers0