5

I'm looking for a better way to do this, if possible:

import subprocess

f = open('temp.file', 'w+')
f.write('hello world')
f.close()

out = subprocess.check_output(['cat', 'temp.file'])

print out

subprocess.check_output(['rm', 'temp.file'])

In this example I'm creating a file and passing it as input to cat (in reality it's not cat I'm running but some other program that parses an input pcap file).

What I'm wondering is, is there a way in Python I can create a 'file-like object' with some content, and pipe this file-like object as input to a command-line program. If it is possible, I reckon it would be more efficient than writing a file to the disk and then deleting that file.

Juicy
  • 11,840
  • 35
  • 123
  • 212
  • Is this actually a bottleneck for your program or are you just assuming it might be at some point? – kylieCatt Jun 17 '15 at 12:06
  • If the recipient program is not configured to read from STDIN, then you don't have a choice but to write to disk; otherwise you could stream the data in by simple redirection - assuming I understood what you are trying to do. – Burhan Khalid Jun 17 '15 at 12:07
  • 1
    @BurhanKhalid: there are [named pipes and /dev/fd/N filenames (fdescfs)](http://stackoverflow.com/a/28840955/4279) that allow you to pass the input as a filename without writing the corresponding content to disk. – jfs Jun 17 '15 at 13:45

4 Answers4

4

check_output takes a stdin input argument to specify a file-like object to connect to the process's standard input.

with open('temp.file') as input:
    out = subprocess.check_output(['cat'], stdin=input)

Also, there's no need to shell out to run rm; you can remove the file directly from Python:

os.remove('temp.file')
chepner
  • 497,756
  • 71
  • 530
  • 681
  • 8
    you can pass a file-like object only if `.fileno()` returns a valid file descriptor e.g., you can't pass `StringIO` object. – jfs Jun 17 '15 at 13:49
3

If the program is configured to read from stdin, you can use Popen.communicate:

>>> from subprocess import Popen, PIPE
>>> p = Popen('cat', stdout=PIPE, stdin=PIPE, stderr=PIPE)
>>> out, err = p.communicate(input=b"Hello world!")
>>> out
'Hello world!'
Vincent
  • 12,919
  • 1
  • 42
  • 64
3

You can write to a TemporaryFile

import subprocess
from tempfile import TemporaryFile
f = TemporaryFile("w")
f.write("foo")
f.seek(0)
out = subprocess.check_output(['cat'],stdin=f)

print(out)
b'foo'

If you just want to write to a file like object and get the content:

from io import StringIO
f = StringIO()
f.write("foo")

print(f.getvalue())
Padraic Cunningham
  • 176,452
  • 29
  • 245
  • 321
3

If the command accepts only filenames, if it doesn't read input from its stdin i.e., if you can't use stdin=PIPE + .communicate() or stdin=real_file then you could try /dev/fd/# filenames:

#!/usr/bin/env python3
import os
import subprocess
import threading

def pump_input(pipe):
    with pipe:
        for i in range(3):
            print(i, file=pipe)

r, w = os.pipe()
try:
    threading.Thread(target=pump_input, args=[open(w, 'w')]).start()
    out = subprocess.check_output(['cat', '/dev/fd/'+str(r)], pass_fds=[r])
finally:
    os.close(r)
print('got:', out)

No content touches the disk. The input is passed to the subprocess via the pipe directly.

If you have a file-like object that is not a real file (otherwise, just pass its name as the command-line argument) then pump_input() could look like:

import shutil

def pump_input(pipe):
    with pipe:
        shutil.copyfileobj(file_like_object, pipe)
Community
  • 1
  • 1
jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • This is fantastic! I generalized this into a fancy context manager that accepts a string, and yields an object that can be passed directly to `pass_fds` as well as placed directly within `args`: https://gist.github.com/theY4Kman/2a80767e9fdabcc4634556c667fa59c8 – theY4Kman Jul 20 '21 at 20:51
  • Great! What about if the output is too large to fit in memory, so I want to stream it to another thread/process? (Or obtain a file-like object to call `.read()` on, which provides back-pressure by blocking writes when full.) – falsePockets Oct 22 '21 at 07:02
  • 1
    @falsePockets write as much data as you like in your `pump_input()`. `check_output` and `cat` are just examples. Use your own command instead of `cat`. If your command produces unlimited output, you can redirect it into another process (the simplest option is to run shell ["data-source | data-sink"](https://stackoverflow.com/q/295459/4279)). If you want to consume data in Python, see [Read streaming input from subprocess.communicate()](https://stackoverflow.com/a/17698359/4279) – jfs Oct 22 '21 at 13:51