python concatenate files using subprocess

Question

I have been searching and although I find long and complicated (many features I do not need) on how to just simply have python have linux use the cat function to concatenate files to a single file.

From my reading apparently subprocess is the way to do this. Here is what I have but obviously does not work :(

subprocess.call("cat", str(myfilelist[0]), str(myfilelist[1]), str(myfilelist[2]), str(myfilelist[3]), ">", "concatinatedfile.txt"])

the above assumes:

myfilelist[]

the above list has 4 filenames + paths as a list; for example one item in the list is "mypath/myfile1.txt"

I would take non subprocess (but simple) methods also

score 4 · Answer 1 · answered Jun 27 '13 at 23:27

4

If you want to use cat and redirection > you must call a shell, for example via system:

from os import system
system("cat a.txt b.txt > c.txt")

However you must pay attention to code injection.

answered Jun 27 '13 at 23:27

Emanuele Paolini

9,912
3
38
64

coolz! this worked faster. so what is the difference of using subprocess then? – StudentOfScience Jun 27 '13 at 23:29
good answer(+1) i just did the same thing with subprocess – Joran Beasley Jun 27 '13 at 23:30
here you are running a shell so it is slower and more prone to security risks – Emanuele Paolini Jun 27 '13 at 23:31

score 4 · Accepted Answer · answered Jun 27 '13 at 23:30

since > is a shell function you need to do shell=True

subprocess.call("echo hello world > some.txt",shell=True) ... works in windows at least

alternatively you can do somethign like

with open("redirected_output.txt") as f:
    subprocess.call("/home/bin/cat some_file.txt",stdout=f)

score 2 · Answer 3 · edited May 23 '17 at 12:31

2

See this question. His solution seems to be concise and easy to understand, I'll post it here:

filenames = ['file1.txt', 'file2.txt', ...]
with open('path/to/output/file', 'w') as outfile:
    for fname in filenames:
        with open(fname) as infile:
            outfile.write(infile.read())

edited May 23 '17 at 12:31

Community

1
1

answered Jun 27 '13 at 23:16

Stephan

16,509
7
35
61

Thanks Stephan. THis does work, and I had seen it. It is just that it is CPU intensive when you have text files each over 10k or more lines :( I wanted to use the faster cat function of linux. – StudentOfScience Jun 27 '13 at 23:18
[this](https://gist.github.com/dimo414/2993381) seems to be some really optimized concatenation @StudentOfScience – Stephan Jun 27 '13 at 23:21
1

I just ran it with my big files, took 1 min 34 sec. When I use cat manually (minus me typing the code) it take 24 sec. that is why. Not that I do not have an additional 1 min and 10 sec to give, but just as a learning opportunity to use subprocess, or python to linux I asked the question. Thx. – StudentOfScience Jun 27 '13 at 23:25
[this](http://stackoverflow.com/a/9882548/2502012) would help you with your learning opportunity to use subprocess for concatenation – Stephan Jun 27 '13 at 23:31

python concatenate files using subprocess

3 Answers3