How to solve directory path problem recursively in Python?

Question

I am doing REST API calls to fetch folders of a SharePoint document library.

I want to recursively get all the folder paths in the entire directory tree.

I have written a function to get a list of sub-folders from a given folder, but am not sure how to traverse till N-th directory and get all folder paths.

Suppose for example, the present SharePoint Document Library structure is as the following JSON (fo=folder; f=file):

{
  "root": [
    {
      "fo1": {
        "fo1": "f1",
        "fo2": ["f1", "f2"]
      },
      "fo2": ["fi1", "fi2"]
    },
    "fi1","fi2"]
}

From the above example, I want a list of paths of all folders/directories: Eg output should be:

["/root/fo1/", "/root/fo1/fo1/", "/root/fo1/fo2/", "/root/fo2/"]

Since it is a REST API call, so I do not know the structure from beforehand until I run a query of get subfolders and then going inside each subfolder to get their respective subfolder.

The present (following) function I wrote is getting me the data till 1 level down (subfolder, since it is inner iteration based and not recursion), how can I achieve a recursion based solution to get all the unique folder paths as a list?

def print_root_contents(ctx):

    try:
        list_object = ctx.web.lists.get_by_title('Documents')
        folder = list_object.root_folder
        ctx.load(folder)
        ctx.execute_query()

        folders = folder.folders
        ctx.load(folders)
        ctx.execute_query()

        for myfolder in folders:
            print("For Folder : {0}".format(myfolder.properties["Name"]))
            folder_list, files_list = print_folder_contents(ctx, myfolder.properties["Name"])
            print("Sub folders - ", folder_list)
            print("Files - ", files_list)

    except Exception as e:
        print('Problem printing out library contents: ', e)


def print_folder_contents(ctx, folder_name):

    try:
        folder = ctx.web.get_folder_by_server_relative_url("/sites/abc/Shared Documents/"+folder_name+"/")
        ctx.load(folder)
        ctx.execute_query()

        # Folders
        fold_names = []
        sub_folders = folder.folders
        ctx.load(sub_folders)
        ctx.execute_query()
        for s_folder in sub_folders:
            # folder_name = folder_name+"/"+s_folder.properties["Name"]
            # print("Folder name: {0}".format(folder.properties["Name"]))
            fold_names.append(s_folder.properties["Name"])

        return fold_names

    except Exception as e:
        print('Problem printing out library contents: ', e)

In the last function (print_folder_contents) above, I am unable to form a recursive logic to keep appending folders and subfolders recursively and stop it when there is no more folders inside the n-th folder and continue for next sibling folder one level up.

Finding it really challenging. Any help?

Answer 1

I know this answer is quite late to the game, but you could perform something like the following to get a flat list of all child SharePoint objects given some parent directory.

This works because we are constantly extending a single list, rather than creating nested objects when leveraging the list.append() method when recursing some directory tree.

I'm sure there would be opportunities to improve the below snippet, but I believe this should help you achieve your objective.

Cheers,

rs311

from office365.sharepoint.client_context import ClientContext


def get_items_in_directory(ctx_client: ClientContext,
                           directory_relative_uri: str,
                           recursive: bool = True):
    """
    This function provides a way to get all items in a directory in SharePoint, with
    the option to traverse nested directories to extract all child objects.
    
    :param ctx_client: office365.sharepoint.client_context.ClientContext object
        SharePoint ClientContext object.
    :param directory_relative_uri: str
        Path to directory in SharePoint. 
    :param recursive: bool
        default = False
        Tells function whether or not to perform a recursive call.
    :return: list
        Returns a flattened array of all child file and/or folder objects
        given some parent directory. All items will be of the following types:
            - office365.sharepoint.file.File
            - office365.sharepoint.folder.Folder
        
    Examples 
    ---------
    All examples assume you've already authenticated with SharePoint per
    documentation found here:
        - https://github.com/vgrem/Office365-REST-Python-Client#examples
        
    Assumed directory structure:
        some_directory/
            my_file.csv
            your_file.xlsx
            sub_directory_one/
                123.docx
                abc.csv
            sub_directory_two/
                xyz.xlsx
    
    directory = 'some_directory'
    # Non-recursive call
    extracted_child_objects = get_items_in_directory(directory)
    # extracted_child_objects would contain (my_file.csv, your_file.xlsx, sub_directory_one/, sub_directory_two/)
    
    
    # Recursive call
    extracted_child_objects = get_items_in_directory(directory, recursive=True)
    # extracted_child_objects would contain (my_file.csv, your_file.xlsx, sub_directory_one/, sub_directory_two/, sub_directory_one/123.docx, sub_directory_one/abc.csv, sub_directory_two/xyz.xlsx)
    
    """
    contents = list()
    folders = ctx_client.web.get_folder_by_server_relative_url(directory_relative_uri).folders
    ctx_client.load(folders)
    ctx_client.execute_query()

    if recursive:
        for folder in folders:
            contents.extend(
                get_items_in_directory(
                    ctx_client=ctx_client,
                    directory_relative_uri=folder.properties['ServerRelativeUrl'],
                    recursive=recursive)
            )

    contents.extend(folders)

    files = ctx_client.web.get_folder_by_server_relative_url(directory_relative_uri).files
    ctx_client.load(files)
    ctx_client.execute_query()

    contents.extend(files)
    return contents

Answer 2

You can use a generator function that iterates through dict items and yields dict keys and yield keys joined with the paths generated from recursive calls, and if given a list, recursively yield what's generated from recursive calls on the list items:

def paths(d):
    def _paths(d):
        if isinstance(d, dict):
            for k, v in d.items():
                yield k + '/'
                for p in _paths(v):
                    yield '/'.join((k, p))
        elif isinstance(d, list):
            for i in d:
                yield from _paths(i)
    return ['/' + p for p in _paths(d)]

so that given:

d = {
  "root": [
    {
      "fo1": {
        "fo1": "f1",
        "fo2": ["f1", "f2"]
      },
      "fo2": ["fi1", "fi2"]
    },
    "fi1","fi2"]
}

paths(d) returns:

['/root/', '/root/fo1/', '/root/fo1/fo1/', '/root/fo1/fo2/', '/root/fo2/']

Note that your expected output should include '/root/' because root folder should be a valid folder as well.