Running a function on multiple files simultaneously with python

Question

i have a specific function that manipulates text files via input of directory and file name.

The defined function is as below

def nav2xy(target_directory, target_file):

after_rows = f'MOD {target_file}_alines.txt'
after_columns = f'MOD {target_file}_acolumns.txt'

# this segment is used to remove top lines(8 in this case) for work with only the actual data
infile = open(f'{target_directory}/{target_file}', 'r').readlines()
with open(after_rows, 'w') as outfile:
    for index, line in enumerate(infile):
        if index >= 8:
            outfile.write(line)

# this segment removes the necessary columns, in this case leaving only coordinates for gmt use
with open(after_rows) as In, open(after_columns, "w") as Out:
    for line in In:
        values = line.split()
        Out.write(f"{values[4]} {values[5]}\n")

i am searching for a way to run this code once on all files in the chosen directory(could be targeted by name or just do all of them), should i change the function to use only the file name?

tried running the function this way, to no avail

for i in os.listdir('Geoseas_related_files'):
    nav2xy('target_directory', i)

this way works perfectly, although somehow i still get this error with it.

    (base) ms-iMac:python gan$ python3 coordinates_fromtxt.py 
Traceback (most recent call last):
  File "coordinates_fromtxt.py", line 7, in <module>
    nav2xy('Geoseas_related_files', str(i))
  File "/Users/gadraifman/research/python/GAD_MSC/Nav.py", line 19, in nav2xy
Out.write(f"{values[4]} {values[5]}\n")

IndexError: list index out of range

any help or advice would be a great help,

Take a look at os.walk: https://docs.python.org/3.5/library/os.html#os.walk — Artog, Aug 08 '19 at 13:03
Do you want to loop a single dir or recursively down a tree? — Artog, Aug 08 '19 at 13:34
sing dir, seems to work with the os.listdir now that i changed file to i(silly mistake), although in addition to working i still get an error as above — gad raifman, Aug 08 '19 at 14:12

score 1 · Accepted Answer · answered Aug 08 '19 at 14:05

From what I gather from Iterating through directories with Python, the best way to loop directories is using glob.

I made some extensive other modifications to your code to simplify it and remove the middle step of saving lines to a file just to read them again. If this step is mandatory, then feel free to add it back.

import os, glob

def nav2xy(target_file):

    # New file name, just appending stuff. 
    # "target_file" will contain the path as defined by root_dir + current filename
    after_columns = f'{target_file}_acolumns.txt'

    with open(target_file, 'r') as infile, open(after_columns, "w") as outfile:
        content = infile.readlines()
        #
        #                    --- Skip 8 lines here
        #                   |
        #                   v
        for line in content[8:]:
            # No need to write the lines to a file, just to read them again.
            # Process directly
            values = line.split()
            outfile.write(f"{values[4]} {values[5]}\n")

# I guess this is the dir you want to loop through. 
# Maybe an absolute path c:\path\to\files is better.
root_dir = 'Geoseas_related_files/*'

for file_or_dir in glob.iglob(os.path.join(root_dir,"*")):
    # Skip directories, if there are any.
    if os.path.isfile(file_or_dir): 
        nav2xy(file_or_dir)

Running a function on multiple files simultaneously with python

The defined function is as below

tried running the function this way, to no avail

1 Answers1