How do I pass a string into subprocess.Popen (using the stdin argument)?

Question

If I do the following:

import subprocess
from cStringIO import StringIO
subprocess.Popen(['grep','f'],stdout=subprocess.PIPE,stdin=StringIO('one\ntwo\nthree\nfour\nfive\nsix\n')).communicate()[0]

I get:

Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/build/toolchain/mac32/python-2.4.3/lib/python2.4/subprocess.py", line 533, in __init__
    (p2cread, p2cwrite,
  File "/build/toolchain/mac32/python-2.4.3/lib/python2.4/subprocess.py", line 830, in _get_handles
    p2cread = stdin.fileno()
AttributeError: 'cStringIO.StringI' object has no attribute 'fileno'

Apparently a cStringIO.StringIO object doesn't quack close enough to a file duck to suit subprocess.Popen. How do I work around this?

Instead of disputing my answer with this being deleted, I'm adding it as a comment... Recommended reading: [Doug Hellmann's Python Module of the Week blog post on subprocess](http://www.doughellmann.com/PyMOTW/subprocess/). — Daryl Spitzer, Jun 18 '13 at 22:43
the blog post contains multiple errors e.g., [the very first code example: `call(['ls', '-1'], shell=True)`](http://www.doughellmann.com/PyMOTW/subprocess/) is incorrect. I recommend to read [common questions from subprocess' tag description](http://stackoverflow.com/tags/subprocess/info) instead. In particular, [Why subprocess.Popen doesn't work when args is sequence?](http://stackoverflow.com/q/2400878/4279) explains why `call(['ls', '-1'], shell=True)` is wrong. I remember leaving comments under the blog post but I don't see them now for some reason. — jfs, Mar 17 '16 at 14:24
For the newer `subprocess.run` see https://stackoverflow.com/questions/48752152/how-do-i-pass-a-string-in-to-subprocess-run-using-stdin-in-python-3/59496029#59496029 — Boris Verkhovskiy, Dec 27 '19 at 04:46

jfs · Accepted Answer · 2020-11-08T06:45:10.237

389

Popen.communicate() documentation:

Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.

Replacing os.popen*

    pipe = os.popen(cmd, 'w', bufsize)
    # ==>
    pipe = Popen(cmd, shell=True, bufsize=bufsize, stdin=PIPE).stdin

Warning Use communicate() rather than stdin.write(), stdout.read() or stderr.read() to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

So your example could be written as follows:

from subprocess import Popen, PIPE, STDOUT

p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)    
grep_stdout = p.communicate(input=b'one\ntwo\nthree\nfour\nfive\nsix\n')[0]
print(grep_stdout.decode())
# -> four
# -> five
# ->

On Python 3.5+ (3.6+ for encoding), you could use subprocess.run, to pass input as a string to an external command and get its exit status, and its output as a string back in one call:

#!/usr/bin/env python3
from subprocess import run, PIPE

p = run(['grep', 'f'], stdout=PIPE,
        input='one\ntwo\nthree\nfour\nfive\nsix\n', encoding='ascii')
print(p.returncode)
# -> 0
print(p.stdout)
# -> four
# -> five
# ->

edited Nov 08 '20 at 06:45

answered Oct 03 '08 at 04:11

jfs

399,953
195
994
1,670

17

This is NOT a good solution. In particular, you cannot asynchronously process p.stdout.readline output if you do this since you'd have to wait for the entire stdout to arrive. It's is also memory-inefficient. – OTZ Aug 20 '10 at 21:59
9

@OTZ What's a better solution? – Nick T Nov 17 '10 at 21:27
13

@Nick T: "*better*" depends on context. Newton's laws are good for the domain they are applicable but you need special relativity to design GPS. See [Non-blocking read on a subprocess.PIPE in python](http://stackoverflow.com/q/375427). – jfs Oct 18 '11 at 20:25
9

But note the NOTE for [communicate](http://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate): "do not use this method if the data size is large or unlimited" – Owen Jan 22 '14 at 00:21
Can someone explain what each step of the commands are doing so that they may be applied to other problems? – LP_640 Feb 25 '17 at 05:25
1

@LP_640 `subprocess.Popen` may be applied to great many problems. You could start with [common problems linked in the subprocess' tag description](http://stackoverflow.com/tags/subprocess/info). – jfs Feb 27 '17 at 21:19
@J.F.Sebastian But when we write, `p.communicate(input=b'one\n')`. I know we are writing to child process's `stdin`. But, Is parent process writing thru it's `stdout` to child process's `stdin`? Can you please explain like [this](https://github.com/shamhub/python_programming/blob/master/3_Concurrency/Website%20checker/scn.jpg), in your answer? – overexchange Jul 13 '17 at 21:33
@overexchange `print` in the code prints to python's stdout. It is unrelated to the passing of input to a subprocess as a bytestring. grep's stdin, stdout, stderr have nothing to do with python's stdin, stdout, stderr in the example. All grep's standard streams are redirected here. .communicate() uses grep's stdin, stdout, stderr in a safe manner (it may use threads, async. io under the hood. It hides the complexity: you just pass a string and it is delivered to the child via pipe for you and the corresponding output is read from another pipe that is connected to grep's stdout and returned). – jfs Jul 13 '17 at 22:25
@J.F.Sebastian I know that, they are subprocess grep's stdin, stdout, stderr is used, when I mean those descriptors in *Popen(grep, stdout, stderr).communicate(b'something')*. My question is, How parent process is sending data(`b'something'`) to grep's stdin? – overexchange Jul 13 '17 at 23:27
@overexchange stdin=PIPE in the code creates a pipe. Any data written by python on one end of the pipe can be read by the grep process on the other end (it is connected to grep's stdin, grep just reads from its stdin). python sees its own end of the pipe as a file-like object `p.stdin` with all the usual methods: .write(), .flush(), .fileno(), .close(). – jfs Jul 14 '17 at 08:06
2

You need python 3.6 to use the `input` arg with `subprocess.run()`. Older versions of python3 work if you do this: `p = run(['grep', 'f'], stdout=PIPE, input=some_string.encode('ascii'))` – TaborKelly Aug 03 '18 at 19:27
1

@TaborKelly: 1- note: you don't need `.encode()` -- the code use `encoding` parameter 3- "current Python 3 version" refered to Python 3.6. It is Python 3.7 now. – jfs Aug 03 '18 at 19:44
Sorry, I have a typo. You need Python 3.6 to use `encoding`, my example works on Python 3.5. You need python 3.6 to use the `encoding` arg with `subprocess.run()`. Which comes in handy since not everyone is running with Python 3.6 yet, for example Debian Stable is on Python 3.5. – TaborKelly Aug 03 '18 at 20:41
Note that the `input` parameter of `subprocess.run` expects a bytes-string unless `encoding` is specified – robertspierre Feb 23 '22 at 04:57

score 50 · Answer 2 · edited Jan 18 '18 at 00:31

50

I figured out this workaround:

>>> p = subprocess.Popen(['grep','f'],stdout=subprocess.PIPE,stdin=subprocess.PIPE)
>>> p.stdin.write(b'one\ntwo\nthree\nfour\nfive\nsix\n') #expects a bytes type object
>>> p.communicate()[0]
'four\nfive\n'
>>> p.stdin.close()

Is there a better one?

edited Jan 18 '18 at 00:31

tobsecret

2,442
15
26

answered Oct 02 '08 at 17:27

Daryl Spitzer

143,156
76
154
173

27

@Moe: `stdin.write()` usage is discouraged, `p.communicate()` should be used. See my answer. – jfs Oct 03 '08 at 14:26
12

Per the subprocess documentation: Warning - Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process. – Jason Mock Aug 25 '10 at 20:49
2

I think this is good way to do it if you're confident that your stdout/err won't ever fill up (for instance, it's going to a file, or another thread is eating it) and you have an unbounded amount of data to be sent to stdin. – Lucretiel May 09 '16 at 19:24
1

In particular, doing it this way still ensures that stdin is closed, so that if the subprocesses is one that consumes input forever, the `communicate` will close the pipe and allow the process to end gracefully. – Lucretiel May 09 '16 at 19:25
@Lucretiel, if the process consumes stdin forever, then presumably it can still write stdout forever, so we'd need completely different techniques all-round (can't `read()` from it, as `communicate()` does even with no arguments). – Charles Duffy Mar 13 '20 at 11:27
@Lucretiel, anyhow, to avoid deadlocks you'd need the `p.stdin.write()` to be done in a different thread, and this answer doesn't show the necessary techniques. `p.stdin.write()` may have a place, but its place is not in an answer that's so short and simple as to not demonstrate how to use it *safely*. – Charles Duffy Mar 13 '20 at 11:28

Flimm · Answer 3 · 2021-12-09T10:43:28.477

38

There's a beautiful solution if you're using Python 3.4 or better. Use the input argument instead of the stdin argument, which accepts a bytes argument:

output_bytes = subprocess.check_output(
    ["sed", "s/foo/bar/"],
    input=b"foo",
)

This works for check_output and run, but not call or check_call for some reason.

In Python 3.7+, you can also add text=True to make check_output take a string as input and return a string (instead of bytes):

output_string = subprocess.check_output(
    ["sed", "s/foo/bar/"],
    input="foo",
    text=True,
)

edited Dec 09 '21 at 10:43

answered Dec 08 '16 at 10:04

Flimm

136,138
45
251
267

7

@vidstige You're right, that's weird. I would consider filing this as an Python bug, I don't see any good reason in why `check_output` should have an `input` argument, but not `call`. – Flimm Oct 05 '17 at 14:35
3

This is the best answer for Python 3.4+ (using it in Python 3.6). It indeed does not work with `check_call` but it works for `run`. It also works with input=string as long as you pass an encoding argument too according to the documentation. – Nikolaos Georgiou Feb 22 '19 at 02:48
@Flimm the reason is obvious: `run` and `check_output` use `communicate` under the hood, while `call` and `check_call` do not. `communicate` is much more heavy, since it involves `select` to work with streams, `call` and `check_call` are much simplier and faster. – Vadim Fint Nov 12 '22 at 15:12

score 31 · Answer 4 · answered Nov 02 '15 at 16:34

31

I'm a bit surprised nobody suggested creating a pipe, which is in my opinion the far simplest way to pass a string to stdin of a subprocess:

read, write = os.pipe()
os.write(write, "stdin input here")
os.close(write)

subprocess.check_call(['your-command'], stdin=read)

answered Nov 02 '15 at 16:34

Graham Christensen

839
7
7

3

The `os` and the `subprocess` documentation both agree that you should prefer the latter over the former. This is a legacy solution which has a (slightly less concise) standard replacement; the accepted answer quotes the pertinent documentation. – tripleee May 04 '16 at 11:53
1

I'm not sure that's correct, tripleee. The quoted documentation says why it is hard to use the pipes created by the process, but in this solution it creates a pipe and passes it in. I believe it avoids the potential deadlock problems of managing the pipes after the process has already started. – Graham Christensen May 07 '16 at 15:09
os.popen is deprecated in favour of subprocess – hd1 Feb 18 '17 at 17:30
3

-1: it leads to the deadlock, it may loose data. This functionality is already provided by the subprocess module. Use it instead of reimplementing it poorly (try to write a value that is larger than an OS pipe buffer) – jfs Aug 18 '17 at 06:24
You deserve the best good man, thank you for the simplest and cleverest solution – Felipe Buccioni Oct 30 '19 at 21:05
2

@tripleee the implementation of pipes in subprocess module is laughably bad, and is impossible to control. You cannot even get the information about the size of the built-in buffer, not to mention, you cannot tell it what are the read and write ends of the pipe, nor can you change the built-in buffer. In not so many words: subprocess pipes are trash. Don't use them. – wvxvw Dec 27 '20 at 13:18

score 15 · Answer 5 · answered Jul 27 '14 at 15:29

15

I am using python3 and found out that you need to encode your string before you can pass it into stdin:

p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=PIPE)
out, err = p.communicate(input='one\ntwo\nthree\nfour\nfive\nsix\n'.encode())
print(out)

answered Jul 27 '14 at 15:29

qed

22,298
21
125
196

5

You don't specifically need to encode the input, it just wants a bytes-like object (e.g. `b'something'`). It will return err and out as bytes also. If you want to avoid this, you can pass `universal_newlines=True` to `Popen`. Then it will accept input as str and will return err/out as str also. – Six Jan 17 '16 at 15:00
2

But beware, `universal_newlines=True` will also convert your newlines to match your system – Nacht Feb 25 '16 at 12:35
1

If you're using Python 3, see [my answer](http://stackoverflow.com/a/41036665/247696) for an even more convenient solution. – Flimm Dec 08 '16 at 10:05

score 12 · Answer 6 · edited Feb 27 '20 at 08:46

12

Apparently a cStringIO.StringIO object doesn't quack close enough to a file duck to suit subprocess.Popen

I'm afraid not. The pipe is a low-level OS concept, so it absolutely requires a file object that is represented by an OS-level file descriptor. Your workaround is the right one.

edited Feb 27 '20 at 08:46

Shumatsu

150
9

answered Oct 02 '08 at 18:33

Dan Lenski

76,929
13
76
124

score 11 · Answer 7 · answered Apr 13 '12 at 03:36

11

from subprocess import Popen, PIPE
from tempfile import SpooledTemporaryFile as tempfile
f = tempfile()
f.write('one\ntwo\nthree\nfour\nfive\nsix\n')
f.seek(0)
print Popen(['/bin/grep','f'],stdout=PIPE,stdin=f).stdout.read()
f.close()

answered Apr 13 '12 at 03:36

Michael Waddell

279
2
6

3

fyi, tempfile.SpooledTemporaryFile.__doc__ says: Temporary file wrapper, specialized to switch from StringIO to a real file when it exceeds a certain size or when a fileno is needed. – Doug F Aug 09 '13 at 20:18

score 7 · Answer 8 · edited Oct 08 '14 at 16:15

7

"""
Ex: Dialog (2-way) with a Popen()
"""

p = subprocess.Popen('Your Command Here',
                 stdout=subprocess.PIPE,
                 stderr=subprocess.STDOUT,
                 stdin=PIPE,
                 shell=True,
                 bufsize=0)
p.stdin.write('START\n')
out = p.stdout.readline()
while out:
  line = out
  line = line.rstrip("\n")

  if "WHATEVER1" in line:
      pr = 1
      p.stdin.write('DO 1\n')
      out = p.stdout.readline()
      continue

  if "WHATEVER2" in line:
      pr = 2
      p.stdin.write('DO 2\n')
      out = p.stdout.readline()
      continue
"""
..........
"""

out = p.stdout.readline()

p.wait()

edited Oct 08 '14 at 16:15

Arne Babenhauserheide

2,423
29
23

answered Jun 14 '13 at 13:20

Lucien Hercaud

97
1
1

5

Because `shell=True` is so commonly used for no good reason, and this is a popular question, let me point out that there are a lot of situations where `Popen(['cmd', 'with', 'args'])` is decidedly better than `Popen('cmd with args', shell=True)` and having the shell break the command and arguments into tokens, but not otherwise providing anything useful, while adding a significant amount of complexity and thus also attack surface. – tripleee Oct 29 '14 at 13:43

score 7 · Answer 9 · answered Dec 27 '19 at 04:29

On Python 3.7+ do this:

my_data = "whatever you want\nshould match this f"
subprocess.run(["grep", "f"], text=True, input=my_data)

and you'll probably want to add capture_output=True to get the output of running the command as a string.

On older versions of Python, replace text=True with universal_newlines=True:

subprocess.run(["grep", "f"], universal_newlines=True, input=my_data)

score 6 · Answer 10 · edited May 23 '17 at 11:33

Beware that Popen.communicate(input=s)may give you trouble ifsis too big, because apparently the parent process will buffer it before forking the child subprocess, meaning it needs "twice as much" used memory at that point (at least according to the "under the hood" explanation and linked documentation found here). In my particular case,swas a generator that was first fully expanded and only then written tostdin so the parent process was huge right before the child was spawned, and no memory was left to fork it:

File "/opt/local/stow/python-2.7.2/lib/python2.7/subprocess.py", line 1130, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory

score 3 · Answer 11 · answered Mar 22 '21 at 21:35

This is overkill for grep, but through my journeys I've learned about the Linux command expect, and the python library pexpect

expect: dialogue with interactive programs
pexpect: Python module for spawning child applications; controlling them; and responding to expected patterns in their output.

import pexpect
child = pexpect.spawn('grep f', timeout=10)
child.sendline('text to match')
print(child.before)

Working with interactive shell applications like ftp is trivial with pexpect

import pexpect
child = pexpect.spawn ('ftp ftp.openbsd.org')
child.expect ('Name .*: ')
child.sendline ('anonymous')
child.expect ('Password:')
child.sendline ('noah@example.com')
child.expect ('ftp> ')
child.sendline ('ls /pub/OpenBSD/')
child.expect ('ftp> ')
print child.before   # Print the result of the ls command.
child.interact()     # Give control of the child to the user.

score 2 · Answer 12 · edited Jun 27 '12 at 14:01

2

p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)    
p.stdin.write('one\n')
time.sleep(0.5)
p.stdin.write('two\n')
time.sleep(0.5)
p.stdin.write('three\n')
time.sleep(0.5)
testresult = p.communicate()[0]
time.sleep(0.5)
print(testresult)

edited Jun 27 '12 at 14:01

dusan

9,104
3
35
55

answered Apr 09 '09 at 04:39

NameError: global name 'PIPE' is not defined – Hayden Thring Apr 11 '23 at 00:59

How do I pass a string into subprocess.Popen (using the stdin argument)?

12 Answers12

Linked

Related