Trying to avoid nested for loops

Question

How do I get rid of these nested for loops in my code? I have tried to use list comprehension but didn't manage to create anything good. Thank you for your help! Here's part of my code:

    folders = [i for i in range(1, int(number_of_folders) + 1)]
    subfolders = [i for i in range(1, int(number_of_subfolders) + 1)]
    files = [i for i in range(1, int(number_of_files) + 1)]        

    for i in folders:
        folderpath = path + "/folder-" + str(i)

        for j in subfolders:
            subfolderpath = folderpath + "/subfolder-" + str(j)
            os.makedirs(subfolderpath)

            for k in files:
                file_path = subfolderpath + "/files-" + str(j) + '-' + str(k) + ".txt"
                open(file_path, "w")

I commented that the above code was reasonably clear and efficient. The one change I would make is to either use generator expressions, for example, `folders = (i for i in range(1, int(number_of_folders) + 1)` or better yet just say `for i in range(1, int(number_of_folders) + 1)` and eliminate the `folders` variable altogether (mostly for clarity and conciseness). But for small values of `number_of_folders`, it probably makes very little difference efficiency wise. — Booboo, Feb 05 '20 at 12:22

Aaron Zolnai-Lucas · Accepted Answer · 2020-02-05T12:22:50.573

Firstly you can cut down on the code in each loop and tidy up with os.path.join:

for i in folders:
        for j in subfolders:
            subfolder_path = os.path.join(path, f"folder{i}", f"subfolder{j}")
            os.makedirs(subfolder_path)
            for k in files:
                file_path = os.path.join(subfolderpath, "files-{j}-{k}.txt")
                open(file_path, "w")

And then the first 2 loops can be made into 1 with itertools.product:

import itertools

for i,j in itertools.product(folders, subfolders):
    subfolder_path = os.path.join(path, f"folder{i}", f"subfolder{j}")
    os.makedirs(subfolder_path)
    for k in files:
         file_path = os.path.join(subfolderpath, "files-{j}-{k}.txt")
         open(file_path, "w")

But how about creating a function to create the file path if it doesn't exist? Then we can condense to a single for loop.

def open_and_create(folder_path, file_name, *a):
    os.makedirs(folder_path, exist_ok=True)
    return open(os.path.join(folder_path, file_name) *a)

for i,j,k in itertools.product(folders, subfolders, files):
    subfolder_path = os.path.join(path, f"folder{i}", f"subfolder{j}")
    open_and_create(subfolder_path, "files-{j}-{k}.txt", 'w')

score 1 · Answer 2 · answered Feb 05 '20 at 11:54

Following code use list comprehensions to form all subfolders and filenames and :

all_subfolders_list = [f"{path}/folder-{str(i)}/subfolder-{str(j)}/"
                  for i in range(1, int(number_of_folders) + 1)
                  for j in range(1, int(number_of_subfolders) + 1)]

all_filenames = [f"{subfolder_path}{k}" for subfolder_path in all_subfolders_list
                 for k in range(1, int(number_of_files) + 1)]

for subfolderpath in all_subfolders_list:
    os.makedirs(subfolderpath)

for file_path in all_filenames:
    open(file_path, "w")

Booboo · Answer 3 · 2020-02-05T12:11:40.040

1

I am not sure why you would want to modify your code to use list comprehensions, for it is reasonably clear and efficient. But a straightforward conversion of your code to use list comprehensions would be:

import os

number_of_folders = 2
number_of_subfolders = 3
number_of_files = 4

path = 'path'

def create_dir_and_file(file_path, file_name):
      os.makdedirs(file_path, exist_ok=True)
      open(file_name, "w")

[create_dir_and_file(f'{path}/folder-{i}/subfolder{j}', f'files{j}-{k}.txt')
      for i in range(1, int(number_of_folders) + 1)
      for j in range(1, int(number_of_subfolders) + 1)
      for k in range(1, int(number_of_files) + 1)]

edited Feb 05 '20 at 12:11

answered Feb 05 '20 at 12:02

Booboo

38,656
3
37
60

I also thought about "compressing" all logic into single list comprehension. However I got error because in this case program tries to create each subfolder `number_of_files` times. – Georgiy Feb 05 '20 at 12:09
Good point. I modified the `os.makedirs` to add `exist_ok=True` to ignore error if the directory already exists. – Booboo Feb 05 '20 at 12:12

Trying to avoid nested for loops

3 Answers3