How to Copy Sub Folders and files into New folder using Python

Question

Very new to iterating over folder structures in python (and python!) All the answers I've found on this site seem very confusing to implement to my situation. Hoping someone can assist.

I have a folder called Downloads. ( 1st level ) This folder is stored at "C:\Users\myusername\Desktop\downloads"

Within this folder I have the following subfolders. (2nd level) Folder path example: C:\Users\myusername\Desktop\downloads\2020-03-13

2020-03-13

2020-03-15... etc

Within each of these 2nd level folders I have another level of folders with pdf files. So if I take 2020-03-13 it has a number of folders below: - 3rd level

22105853

22108288

22182889

Example path for third level: C:\Users\myusername\Desktop\downloads\2020-03-13\22105853

All I am trying to do is create a new folder at the Downloads (1st)level and copy all the folders at the third level into it. Eliminating the second level structure basically.

Desired result. C:\Users\myusername\Desktop\r3\downloads\NEWFOLDER\22105853 C:\Users\myusername\Desktop\r3\downloads\NEWFOLDER\22108288 C:\Users\myusername\Desktop\r3\downloads\NEWFOLDER\22182889

I started the code below and managed to recreate the file structure within a new file called Downloads: But stuck now and hoping someone can help me.

save_dir='C:\\Users\\myusername\\Desktop\\downloads\\'
localpath = os.path.join(save_dir, 'Repository')
if not os.path.exists(localpath):
    try:
        os.mkdir(localpath, mode=777)
        print('MAKE_DIR: ' + localpath)
    except OSError:
        print("directory error occurred")

for root, dirs, files in os.walk(save_dir):
    for dir in dirs:
        path = os.path.join(localpath, dir)
        if '-' not in path and  not os.path.exists(path): 
        #(Checking for '-' to not create folders at sceond level)
            os.mkdir(path, mode=777)
            print(path)

Some remarks: Identifying the path level by characters that you expect on that level is fairly error-prone since you cannot guarantee that the character does not occur on other levels. It may work now, but it's not future-proof. Also note that you are using `dir` as a variable, which is already a built-in-name. This can cause weird bugs. I'd advise you use an IDE like Pycharm and make it a habit to fix all warnings the live code check displays. This act alone will improve your code quality and python skills while you work. As for a solution, the one by @SdahlSean seems a good starting point. — Cerno, Feb 09 '21 at 22:28
Does this answer your question? [Copy file or directories recursively in Python](https://stackoverflow.com/questions/1994488/copy-file-or-directories-recursively-in-python) — Iron Fist, Feb 09 '21 at 23:42
https://docs.python.org/3/library/shutil.html#shutil.copytree — Iron Fist, Feb 09 '21 at 23:42
Yeah I planned to update the search for folders by using a regex or glob or fname matching. it was a stop gap measure. — R M, Feb 10 '21 at 18:21

SdahlSean · Accepted Answer · 2021-02-10T16:39:39.053

0

This code snippet should work:

import os
from distutils.dir_util import copy_tree

root_dir = 'path/to/your/rootdir'
try:
    os.mkdir('path/to/your/rootdir/dirname')
except:
    pass
for folder_name in os.listdir(root_dir):
    path = root_dir + folder_name
    for folder_name in os.listdir(path):
        copy_tree(path + folder_name, 'path/to/your/rootdir/dirname')

just replace the directory names with the names you need

edited Feb 10 '21 at 16:39

answered Feb 09 '21 at 20:53

SdahlSean

573
3
13

This code does not seem to run out of the box. You are not connecting root_dir and folder_name properly (use `os.path.join()` instead) and your try does not have an except. The idea should work though, iterating through all directories on level 1, 2 and 3 using nested for loops, then constructing a target directory structure using only the directory names from levels 1 and 3. – Cerno Feb 09 '21 at 22:16
Thanks! I'll try this out and give feedback. – R M Feb 10 '21 at 18:19
The code didn't work out of the box, but it came pretty close. I was able to come up with what I needed. Thanks!! – R M Feb 10 '21 at 20:19

Umb · Answer 2 · 2021-02-12T00:12:40.247

Using copy_tree is probably the best way to do it, however I prefer check if there are strange files or folders in wrong place and then create folders or copy the files. Here is another way to do that. However be careful if you will create the repository folder inside the root folder and than you will iterate over the root folder, in listdir you will have also the Repository folder.

import os
import shutil
def main_copy():
    save_dir='C:\\Users\\myusername\\Desktop\\downloads'
    localpath = os.path.join(save_dir, 'Repository')
    if not os.path.exists(localpath):
        try:
            os.mkdir(localpath, mode=777)
            print('MAKE_DIR: ' + localpath)
        except OSError:
            print("directory error occurred")
            return
            

    for first_level in os.listdir(save_dir):
        subffirstlevel = os.path.join(save_dir, first_level)
        # skip repository folder
        if subffirstlevel == localpath: continue
        # skip eventually files
        if os.path.isfile(subffirstlevel): continue 
        for folder_name in os.listdir(subffirstlevel): 
            subf = os.path.join(subffirstlevel, folder_name)
            # skip eventually files
            if os.path.isfile(subf): continue 
            newsubf = os.path.join(localpath, folder_name)
            if not os.path.exists(newsubf):
                try:
                    os.mkdir(newsubf, mode=777)
                    print('MAKE_DIR: ' + newsubf)
                except OSError:
                    print("directory error occurred")
                    continue
            for file_name in os.listdir(subf):
                filename = os.path.join(subf, file_name) 
                if os.path.isfile(filename):
                    shutil.copy(filename, os.path.join(newsubf, file_name))
                    print("copy ", file_name)

Thanks for the response, I'll take a look at both solutions. Appreciate it! — R M, Feb 10 '21 at 18:24
I can see where you were going with this, but the code does not go to the right third folder level. I was able to use a bit of your code to form a solution though. — R M, Feb 10 '21 at 20:16
@RM Yeah, You are right, sorry. It is necessary one more for loop, I have edited the code. About my solution, this can work only if the folder will not change. If for some reason sometimes in future you will have more levels, this could be not the right solution. — Umb, Feb 12 '21 at 00:13

How to Copy Sub Folders and files into New folder using Python

2 Answers2