Python: Reading multiple files and storing the output for a particular file

Question

I have a thousand .xvg files in a directory which I need to read and store an output for each of them. Currently I have a python code which works for only one file. Could you please suggest how do I read all files at once and get an output and store it for every file separately?

    f = open('/home/abc/xyz/coord/coord_1.xvg')
    dat = f.readlines()
    dat1 = dat[22:len(dat)]
    dat2=[]

    for k in dat1:
        dat2.append(k.split())

    for k in dat2:
        if float(k[1])>=9.5:
            print('P')
            break
        elif float(k[1])<=5.9:
            print('R')
            break
        else:
            print('N')

Could you explain what do you mean with *read all files at once*? And show a minimum reproducible example? — DarK_FirefoX, Mar 30 '20 at 15:08
Do you just mean to loop through all the files in a directory with the suffix '.xvg' and apply your code to each file with the change being we write a new corresponding file rather than printing to the console? — DarrylG, Mar 30 '20 at 15:14
@DarK_FirefoX, I have a 1000 files in a directory all of the same kind named coord_1.xvg, coord_2.xvg and so on. I want to read them one by one and store an output for each of them separately. The code I used gives me the output for one file at a time and I'm having to change the file name everytime I do this which is tedious. — accidental_coder, Mar 30 '20 at 15:18
@DarrylG, yes I want new corresponding files storing the output for each of these files. Also .xvg are the only kind of files that exist in this directory so I want to scan all of them and not worry about any other files. — accidental_coder, Mar 30 '20 at 15:20
@accidental_coder--is my refactoring code correct in my answer in assuming you're only processing the 23rd line of each file? — DarrylG, Mar 30 '20 at 16:07

DarrylG · Accepted Answer · 2020-03-30T16:05:10.200

Here's a version but used as much as code as possible to make it easier to follow.

import os

def process_files():
  " Will process all files in folder using your code "
  for file in os.listdir("."): # '.' correspond to the current  directory
                               # You can specify whatever directory, 
                               #such as /usr/accidental_coder/working
    if file.endswith(".xvg"):
        # Find found
        # Output will be with same name but with .txt suffix
        with open(os.path.join(".", file), 'r') as infile, \
          open(os.path.join(".", file.replace('.xvg', '.txt')), 'w') as ofile:

          # Using your original code
          # left alone so you could know how to change if desired
          # (note: your can be shortened)
          dat = infile.readlines()
          dat1 = dat[22:len(dat)]
          dat2=[]

          for k in dat1:
              dat2.append(k.split())
          for k in dat2:
              if float(k[1])>=9.5:
                  ofile.write('P\n')
                  break
              elif float(k[1])<=5.9:
                  ofile.write('R\n')
                  break
              else:
                  ofile.write('N\n')

process_files()

Refactoring Your Code for Better Performance

Seems you just process the 23'rd line in each file

import os

def process_files():
  for file in os.listdir("."):
    # Examples of getting files from directories
    # https://stackoverflow.com/questions/3964681/find-all-files-in-a-directory-with-extension-txt-in-python
    if file.endswith(".xvg"):
        with open(os.path.join(".", file), 'r') as infile, \
          open(os.path.join(".", file.replace('.xvg', '.txt')), 'w') as ofile:

          # Skip first 22 lines
          for _ in range(22):
            next(infile)

          # use data on 23rd line
          data = next(infile)  
          k = data.split()

          if float(k[1])>=9.5:
              ofile.write('P\n')
          elif float(k[1])<=5.9:
              ofile.write('R\n')
          else:
              ofile.write('N\n')

process_files()

Python: Reading multiple files and storing the output for a particular file

1 Answers1