Python doit: Create subtasks using file generated from previous task

Question

With Python doit I want to generate sub-tasks by reading a list of files from a file (that was generated in an earlier task), then yielding actions for each of those files:

def task_generates_list_of_files():

    def generate_file():

        with open('list_of_files.txt', 'w') as fo:

            for x in range(20):
                fo.write(f'thing_{x}.txt\n')

    return {
        'actions': [generate_file],
        'targets': ['list_of_files.txt']
    }

def task_generate_subtasks_using_file():

    with open('list_of_files.txt', 'r') as fo:

        for filename in fo:

            filename = filename.strip()

            yield {
                'name': filename,
                'actions': [f'program_to_run {filename}'],
                'file_dep': ['list_of_files.txt']
            }

However, because list_of_files.txt doesn't exist when the tasks are being set up by doit, it raises a FileNotFoundError exception.

I have seen this answer but am not clear that getargs can work when generating subtasks, because I wouldn't be able to loop through the list of files until they were injected into a Python action, at which point I can't yield any sub-tasks. Doing that results in:

Python Task error:...
It must return:
False for failed task.
True, None, string or dict for successful task
returned <generator object...

I understand that, but the problem is specific to Python `doit` (http://pydoit.org/index.html) where you can make task functions returning a dictionary with tasks for doit to run, or use `yield` to make "sub-tasks". However I can't open my `list_of_files.txt` in the task definitions that are run when `doit` runs the script, because it doesn't exist until the first task is run. The second part was explaining that I can't use a method to pass values between `doit` tasks (called `getargs`) because I can't `yield` subtasks from a function inside a task (http://pydoit.org/dependencies.html#getargs) — Harry, Apr 24 '17 at 16:25
See [_How to create a **Minimal**, Complete, and Verifiable example_](https://stackoverflow.com/help/mcve) that reproduces the problem, then someone may be able to show you how to use `getargs` to accomplish what you want to do. — martineau, Apr 24 '17 at 17:04
The example I have given is a complete example. `pip3 install doit`, then place that code in a `dodo.py`, then run `doit` in the folder and it produces the same error. The tags on this don't help because it's not so much a Python question as a `doit` question, and I don't have enough rep to create a new tag. I appreciate what you're saying, and as Python it won't do anything except define two functions, but using `doit` those `task_` functions are run to define tasks, and fail because the second task definition is run before the first task, but depends on output from the first task. — Harry, Apr 24 '17 at 23:12

score 1 · Accepted Answer · answered Apr 25 '17 at 10:23

In the end, I'd missed something I hadn't seen before in the doit documentation: delayed tasks. You can use create_after to defer creation of a task until after a given task is executed.

In my case this allows list_of_files.txt to be generated by task_generates_list_of_files before list_of_files.txt is read within the task definition of task_generate_subtasks_using_file:

from doit import create_after

# Show output
DOIT_CONFIG = {
    'verbosity': 2
}

def task_generates_list_of_files():

    def generate_file():

        with open('list_of_files.txt', 'w') as fo:

            for x in range(20):
                fo.write(f'thing_{x}.txt\n')

    return {
        'actions': [generate_file],
        'targets': ['list_of_files.txt']
    }

@create_after('generates_list_of_files')
def task_generate_subtasks_using_file():

    with open('list_of_files.txt', 'r') as fo:

        for filename in fo:

            filename = filename.strip()

            yield {
                'name': filename,
                'actions': [f'echo {filename}'],
                'file_dep': ['list_of_files.txt']
            }

Python doit: Create subtasks using file generated from previous task

1 Answers1