Elegant way of combining files - Python

Question

Hi just a quick question. I have a concatenation of files that works flawlessly, but its a bit of a mess. I wondered if there was just a more elegant way to write this:

path = path/to/file/location
with open(path + 'result.txt', 'w') as result, \
        open(path + 'file1.txt') as f1, \
            open(path + 'file2.txt' ) as f2, \
                open(path + 'file3.txt' ) as f3, \
                    open(path + 'file4.txt' ) as f4, \
                        open(path + 'file5.txt' ) as f5, \
                            open(path + 'file6.txt' ) as f6, \
                                open(path + 'file7.txt' ) as f7, \
                                    open(path + 'file8.txt' ) as f8, \
                                        open(path + 'file9.txt' ) as f9, \
                                            open(path + 'file10.txt' ) as f10, \
                                                open(path + 'file11.txt' ) as f11, \
                                                    open(path + 'file12.txt' ) as f12, \
                                                        open(path + 'file13.txt' ) as f13, \
                                                            open(path + 'file14.txt' ) as f14, \
                                                                open(path + 'file15.txt' ) as f15, \
                                                                    open(path + 'file16.txt' ) as f16:
    for line1, line2, line3, line4, line5, line6, line7, line8, \ 
        line9, line10, line11, line12, line13, line14, line15, line16 \
        in zip(f1,f2,f3,f4,f5,f6,f7,f8,f9,f10,f11,f12,f13,f14,f15,f16):

        result.write('{}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, \
        {}, {}, {}\n'.format(line1.rstrip(), line2.rstrip(), line3.rstrip(), line4.rstrip(), \
        line5.rstrip(), line6.rstrip(), line7.rstrip(), line8.rstrip(), line9.rstrip(), \
        line10.rstrip(), line11.rstrip(), line12.rstrip(), line13.rstrip(), line14.rstrip(), \
        line15.rstrip(), line16.rstrip()))

Thanks

Not sure whether it's possible with `with`, but you can certainly put the file names in a list (or just use a `range`) and `open` and `.close` the files manually in a loop. — tobias_k, May 03 '16 at 13:32
See [this question](http://stackoverflow.com/a/3025119/2399799). — dan-man, May 03 '16 at 13:34
ahhh sorry it is part of a function so the `with` works in my script. This is just the interesting bit. — ben, May 03 '16 at 13:34
For the `with` you can use [`contextlib.ExitStack`](https://docs.python.org/2/library/contextlib.html#contextlib.nested) (if you are on Python 3), also described in my answer linked by dan-man. For the rest you need to use lists and loops. — interjay, May 03 '16 at 13:36
@ben - you might want to edit the title of this question to make it more specific to your usage case: "concatenation" is not a terribly helpful way of describing what you are doing. — dan-man, May 03 '16 at 13:39
Is the point to interleave lines from each of these files? Do they all have multiple lines? Do they all have the same number of lines? — Daniel Roseman, May 03 '16 at 13:41
Yes they all have the same number of lines and they all have multiple lines. I want them to write the first line of all the files next to each other on the same line and then the second etc. Like i say, the code works perfectly, I was just looking for a more elegant solution. — ben, May 03 '16 at 13:42

tobias_k · Accepted Answer · 2016-05-03T14:08:26.940

You could always do it without with, putting the files in a list and closing them manually, in a loop, when everything is done. This will also make the format line much simpler:

path = "path/to/file/location/"
with open(path + 'result.txt', 'w') as result:
    files = [open(path + 'file%d.txt' % (n+1)) for n in range(16)]
    form = ", ".join('{}' for f in files) + '\n'
    for lines in zip(*files):
        result.write(form.format(*map(str.rstrip, lines)))
    for f in files:
        f.close()

Or using contextlib.ExitStack, as suggested in comments. This way, the opened files are passed to the stack and that will take care of closing the files after the with block.

with open(path + 'result.txt', 'w') as result, contextlib.ExitStack() as stack:
    files = [stack.enter_context(open(path + 'file%d.txt' % (n+1))) for n in range(16)]
    form = ", ".join('{}' for f in files) + '\n'
    for lines in zip(*files):
        result.write(form.format(*map(str.rstrip, lines)))

score 0 · Answer 2 · answered May 03 '16 at 13:46

You could process the files serially (which gets around the potential problem of having too many files open at once):

result_names = ['result1','result2']
result_index = 0
old_result_path = path + "file1.txt"
for n in xrange(2,17):
    new_result_path = path + (result_names[result_index] if n<16 else 'result.txt')
    input_path = path + "file%d.txt" % n
    with open( old_result_path, 'r' ) as old_input, \
        open( input_path, 'r' ) as new_input, 
            open( new_result_path, 'w' ) as result:
       for line1, line2 in zip( old_input, new_input ):
           result.write('{}, {}\n'.format(line1.rstrip(), line2.rstrip())
    old_result_path = new_result_path
    result_index = 1 - result_index

This will leave result1.txt and result2.txt lying about, which you may or may not care about cleaning up.

Elegant way of combining files - Python

2 Answers2