Deploy files to a directory with a custom structure in Python

Question

I have number of file objects which I would like to redeploy to a directory with a new structure based on requirements stated by the user.

As example could be having these file objects:

1)root\name\date\type\filename
2)root\name\date\type\filename
3)root\name\date\type\filename

...that I want to save (or a copy of them) in a new structure like below after the user defined a need to split type->date->name:

1)root\type\date\name\filename
2)root\type\date\name\filename
3)root\type\date\name\filename

... or even losing levels such as:

1)root\type\filename
2)root\type\filename
3)root\type\filename

I can only come up with the option to go the long way round, taking the initial list and though a process of filtering simply deploy in the new calculated folder structure using basic string operations.

I feel though that someone has probably done this before in a smart way and potentially that a library/module already exists to do this. Does anyone have any ideas?

score 0 · Answer 1 · edited May 23 '17 at 11:53

If you are in UNIX environment, the simpler way to achieve this will be using shell script with cp command.

For example, for copying all files from: /root/name/date/type/filename as /root/date/filename; you need to just do:

cp /root/*/date/*/filename /root/date/filename

OR, if you want to move the file, use mv command:

mv /root/*/date/*/filename /root/date/filename

You may run these commands via Python as well using os.system() as:

import os
os.system("cp \root\*\date\*\filename root\date\filename")

For details, check: Calling an external command in Python

Edit based on comment. For copying /root/name/date/type/filename into /root/date/name/type/filename, you need to just do:

cp /root/name/date/type/filename /root/date/name/type/filename

But make sure that directory /root/date/name/type exists before doing it. In order to make sure it exists, and if not create a directory by using mkdir with -p option as:

mkdir -p /root/date/name/type

OK this could work but i still have questions. What are the * for in your example above? How would you use this method to turn /root/name/date/type/filename into /root/date/name/type/filename? — user3535074, Dec 01 '16 at 22:17

score 0 · Answer 2 · answered Dec 01 '16 at 22:38

Here is a solution using Python glob:

The current levels are: name, date, type and filename:

curr_levels = "name\\date\\type\\filename"
curr_levels = curr_levels.split("\\")

The user want other levels: type, date, name and filename:

user_levels = "type\\date\\name\\filename"
user_levels = user_levels.split("\\")

We can use glob.iglob to iterate the tree structure on 4 levels. The glob pattern is something like: <src_dir>\*\*\*\* (but, we use a more generic way here).

The user structure can be defined with a simple string format. For instance: {type}\{date}\{name}\{filename} on Windows. Wee need to create the directory structure first and then copy (or move) the file.

pattern = os.path.join(source_dir, *("*" * len(curr_levels)))
fmt = os.sep.join(['{{{key}}}'.format(key=key) for key in user_levels])

for source_path in glob.iglob(pattern):
    source_relpath = os.path.relpath(source_path, source_dir)
    parts = source_relpath.split(os.sep)
    values = dict(zip(curr_levels, parts))
    target_relpath = fmt.format(**values)
    target_path = os.path.join(target_dir, target_relpath)
    parent_dir = os.path.dirname(target_path)
    if not os.path.exists(parent_dir):
        os.makedirs(parent_dir)
    shutil.copy2(source_path, target_path)

Note: if your source_dir is the same as target_dir (the root in your question), you need to replace glob.iglob by glob.glob in order to store the whole list of files in memory before processing. This is required to avoid glob.iglob browse the directory tree you are creating…

Deploy files to a directory with a custom structure in Python

2 Answers2