Merging different text files located in different folders using Python

Question

I'm trying to merge content from different .dat files using Python. These files all have the same name:

taueq_polymer_substratechain1600_12_raf2_000_B0_20_S5_0.dat

but are inside different folders which contains other .dat files.

The content of the files is the following form: File Content ( two columns).

I am trying to merge all of these files into a single text file where every every two columns will be next to each other. Something similar to this:Desired Output but in text file.

I found some help here: How to merge content from files with same name but in different folders by Python?

However using this code:

import os

# create a dictionary with file names as keys
# and for each file name the paths where they
# were found
file_paths = {}
for root, dirs, files in os.walk('.'):
    for f in files:
        if f.startswith('taueq_polymer'):
            if f not in file_paths:
                file_paths[f] = []
            file_paths[f].append(root)

# for each file in the dictionary, concatenate
# the content of the files in each directory
# and write the merged content into a file
# with the same name at the top directory
for f, paths in file_paths.items():
    txt = []
    for p in paths:
        with open(os.path.join(p, f)) as f2:
            txt.append(f2.read())
    with open(f, 'w') as f3:
        f3.write(''.join(txt))

The output text file append the data of the files at the bottom of the original file and not next to it. Can anyone tell me how to stack the columns next to each other?

Thank you

Possible duplicate of [Join txt files side by side in python](https://stackoverflow.com/questions/13123657/join-txt-files-side-by-side-in-python) — d_kennetz, Jun 24 '19 at 20:06

score 0 · Accepted Answer · answered Jun 24 '19 at 20:49

file1.txt

1.2 1.2
1.3 1.3
1.3 1.3

file2.txt

8.2 8.2
8.3 8.3
8.3 8.3

The result you except is :

out.txt

1.2 1.2 8.2 8.2
1.3 1.3 8.3 8.3
1.3 1.3 8.3 8.3

So you must read the files line by line and concatenate the lines between them.

paths = 'file1.txt', 'file2.txt'
txt_lines = []
for p in paths:
    with open(p) as f:
        # Iterate through lines.
        for i, line in enumerate(f):
            if line.endswith("\n"):
                # remove the trailing newline
                line = line[:-1]
            try:
                # Concat the line with previous ones of the same index
                txt_lines[i] += ' ' + line
            except IndexError:
                # If the index not exists, append the line to the list
                txt_lines.append(line)
with open('out.txt', 'w') as f:
    # Join each line
    f.write('\n'.join(txt_lines))

Thank you, how would I change the first line so that the program will go into the various folders and get the right text files? — Alessandro Perego, Jun 24 '19 at 21:08
You can continue to do as you did in your code. I simplified the example because I think you had no problem with the rest. — cdrom, Jun 24 '19 at 21:14

Merging different text files located in different folders using Python

1 Answers1