merging big binary files using python 3

Question

How can I merge a few big files into 1 using python 3? it should work like the bash command

cat * > outfile

but it should work on Linux, Windows and OS X.

If I use

outfile = open("outfile", "wb")
for file in os.listdir():
    outfile.write(file.read())

it uses too much RAM

have you tried the "For large files" from here: http://stackoverflow.com/a/13613375/3430986 ? — orestisf, Oct 31 '14 at 18:29
possible duplicate of [Python concatenate text files](http://stackoverflow.com/questions/13613336/python-concatenate-text-files) — inspectorG4dget, Oct 31 '14 at 18:31
`It only works for text files` <- open with `'rb'` - the `b` flag is for binary files — inspectorG4dget, Oct 31 '14 at 18:32

score 2 · Answer 1 · answered Oct 31 '14 at 21:59

For large binary files, instead of reading lines, read chunks that are a multiple of the disk block size. Something like (untested)

BLOCKSIZE = 4096  # typical, I believe
BLOCKS = 1024  # somewhat arbitrary
chunk = BLOCKS * BLOCKSIZE
with open("outfile", "wb") as outfile:
    for fname in os.listdir():
        with open('fname', "rb") as infile
            outfile.write(infile.read(chunk))

merging big binary files using python 3

1 Answers1