1

I am trying to concatenate many different log files that have the same name and are located in different directories. The filenames are in the form of fit_objectname.log where objectname varies. Since I have many pairs of these, I am attempting to accomplish this in a for loop within python, like this:

import re
import glob
from subprocess import Popen, PIPE, STDOUT

new_logfiles = glob.glob('/Users/myusername/Desktop/subdir1/fit*.log')
for log in new_logfiles:
    filename = re.search(r'(fit.+?\.log)', log)[1]
    cmd = "cat /Users/myusername/Desktop/subdir1/{} /Users/myusername/Desktop/subdir2/{} >> {}_concat.log".format(filename, filename, filename[:-4])
    p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)

However, the concatenation is not working. The result for each fit_objectname_concat.log file saved is just the contents of the log file in subdir1.

I've isolated my problem to the loop, since if I run this command on just one file pair (either in terminal or in the python script), it works:

cmd = "cat /Users/myusername/Desktop/subdir1/fit_obj1.log /Users/myusername/Desktop/subdir2/fit_obj1.log > fit_obj1_concat.log"
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)

I've also tried adding a time.sleep(2) statement after each iteration in the loop, but I have the same issue. Any idea what could be going on here?

curious_cosmo
  • 1,184
  • 1
  • 18
  • 36
  • If you want to use cat why not just use a shell script? Or conversely, if you want to use python why not just use python? Why are you extracting the filename out of a path only to printf it right back in? Why are you using regex instead of `os.path` in the first place? This all seems rather Rube-Goldberg-esque. – Jared Smith Jan 30 '19 at 02:15
  • This probably isn't a dup as it doesn't answer your question, but try [using `shutil.copyfileobj()` instead](https://stackoverflow.com/a/27077437/1270789). – Ken Y-N Jan 30 '19 at 02:22

0 Answers0