132

What I do in the command line:

cat file1 file2 file3 > myfile

What I want to do with python:

import subprocess, shlex
my_cmd = 'cat file1 file2 file3 > myfile'
args = shlex.split(my_cmd)
subprocess.call(args) # spits the output in the window i call my python program
jww
  • 97,681
  • 90
  • 411
  • 885
catatemypythoncode
  • 1,323
  • 2
  • 9
  • 4
  • Executing such a command in subprocess would not give you any output. May be you want to run it without _> myfile_ redirecting output from _cat file1 file2 file3_ into python? – PoltoS Feb 11 '11 at 02:52
  • @PoltoS I want to join some files and then process the resulting file. I thought using cat was the easiest alternative. Is there a better/pythonic way to do it? – catatemypythoncode Feb 11 '11 at 02:55
  • `os.sendfile()`-based solution is possible, see [Reproduce the unix cat command in python](http://stackoverflow.com/q/11532980/4279) – jfs May 12 '15 at 18:12
  • 1
    I think that output redirection ('>' or '>>') doesn't work in subprocess.Popen (at least in Python 2.7) (in shell=True mode) In this example, as others point out, you can work around this by not using redirection, but in other cases redirection is useful. If redirection or piping is not supported in subprocess.Popen is should be documented (and/or os.system() should be not be deprecated until this is fixed) – Ribo Jul 19 '16 at 12:50

6 Answers6

364

In Python 3.5+ to redirect the output, just pass an open file handle for the stdout argument to subprocess.run:

# Use a list of args instead of a string
input_files = ['file1', 'file2', 'file3']
my_cmd = ['cat'] + input_files
with open('myfile', "w") as outfile:
    subprocess.run(my_cmd, stdout=outfile)

As others have pointed out, the use of an external command like cat for this purpose is completely extraneous.

Greg Dubicki
  • 5,983
  • 3
  • 55
  • 68
Ryan C. Thompson
  • 40,856
  • 28
  • 97
  • 159
17

UPDATE: os.system is discouraged, albeit still available in Python 3.

Use os.system:

os.system(my_cmd)

If you really want to use subprocess, here's the solution (mostly lifted from the documentation for subprocess):

p = subprocess.Popen(my_cmd, shell=True)
os.waitpid(p.pid, 0)

OTOH, you can avoid system calls entirely:

import shutil

with open('myfile', 'w') as outfile:
    for infile in ('file1', 'file2', 'file3'):
        shutil.copyfileobj(open(infile), outfile)
Marcelo Cantos
  • 181,030
  • 38
  • 327
  • 365
  • 1
    It works, but let me ask you then: What's the point of the subprocess library if os.system already gets the job done? I get the feeling I should've been using subprocess instead since it's a library dedicated to this task, although since I'm doing this just for myself I'll be fine using os.system this time. – catatemypythoncode Feb 11 '11 at 02:53
  • The subprocess library is much more flexible than `os.system`, and can model `os.system` precisely, but it is also more complex to work with. – Marcelo Cantos Feb 11 '11 at 02:56
  • 13
    `os.system` came before `subprocess`. The former is a legacy API that the latter intends to replace. – Santa Feb 11 '11 at 03:27
  • 5
    @catatemypythoncode: you should not use `os.system()` or `shell=True`. To redirect output of a subprocess, use `stdout` parameter as shown in [Ryan Thompson's answer](http://stackoverflow.com/a/6482200/4279). Though you don't need a subprocess (`cat`) in your case, you could concatenate files using pure Python. – jfs May 25 '15 at 07:05
  • Hmm, it's my understanding the usage of os.sytem() is now deprecated – LuckyLuc Oct 16 '15 at 17:02
  • 5
    OTOH = On the other hand – Cephlin Mar 09 '16 at 17:49
  • It's not deprecated. There are valid uses. BUT: if you use untrusted input, validate the heck out of it and then *DON'T* use os.system anyway :D Corollary: if you trust your string (say, it's completely static), then os.system is perfectly fine. – Jürgen A. Erhard Apr 07 '17 at 07:09
5

@PoltoS I want to join some files and then process the resulting file. I thought using cat was the easiest alternative. Is there a better/pythonic way to do it?

Of course: 

with open('myfile', 'w') as outfile:
    for infilename in ['file1', 'file2', 'file3']:
        with open(infilename) as infile:
            outfile.write(infile.read())
SingleNegationElimination
  • 151,563
  • 33
  • 264
  • 304
2
size = 'ffprobe -v error -show_entries format=size -of default=noprint_wrappers=1:nokey=1 dump.mp4 > file'
proc = subprocess.Popen(shlex.split(size), shell=True)
time.sleep(1)
proc.terminate() #proc.kill() modify it by a suggestion
size = ""
with open('file', 'r') as infile:
    for line in infile.readlines():
        size += line.strip()

print(size)
os.remove('file')

When you use subprocess , the process must be killed.This is an example.If you don't kill the process , file will be empty and you can read nothing.It can run on Windows.I can`t make sure that it can run on Unix.

wyx
  • 3,334
  • 6
  • 24
  • 44
  • 1
    It is a bad code example (it won't work on Unix; it demonstrates bad practices `for line in .readlines():`, `s +=`) and `proc.kill()` may lead to information loss in general (it doesn't allow the subprocess to terminate gracefully (on Unix) -- unflushed content is lost). Anyway, the note about buffering is more appropriate as a comment. – jfs Oct 09 '15 at 09:07
  • I run it on Windows is OK(Because kill is equal to terminate on Windows). On Unix may be you should use proc.terminate(). @ J.F. Sebastian I haven`t Unix System on my computer. – wyx Oct 10 '15 at 09:58
  • If you are on Windows then drop `shlex.split()`, drop `shell=True`, drop `>file`, drop `open()`, etc and use `stdout=PIPE`, `Timer(1, proc.terminate).start()`; `output = proc.communicate()[0]` instead. Here's [complete example](http://stackoverflow.com/a/27995163/4279). More solutions: [Stop reading process output in Python without hang?](http://stackoverflow.com/q/4417962/4279) Note: there is no requirement in the question that you need to terminate the child process manually -- you could address other issues e.g., a process may behave differently if its stdout is a tty but it is off-topic. – jfs Oct 10 '15 at 10:04
1

One interesting case would be to update a file by appending similar file to it. Then one would not have to create a new file in the process. It is particularly useful in the case where a large file need to be appended. Here is one possibility using teminal command line directly from python.

import subprocess32 as sub

with open("A.csv","a") as f:
    f.flush()
    sub.Popen(["cat","temp.csv"],stdout=f)
DJJ
  • 2,481
  • 2
  • 28
  • 53
1

It will work if your args will look like ['sh', '-c', 'cat file1 file2 file3 > myfile'] it will mean that output of cat won't pass Python and spawn in shell instead (insted of sh -c you can use bash -c)

Sem
  • 11
  • 2