0

Suppose you have the following project structure:

.
├── Makefile
└── src
    └── 1.py

The program 1.py creates multiple (0, 1, or more) files in the directory build/1. This generalizes to arbitrary numbers, i.e. a program x.py where x is some natural number would create multiple files in the directory build/x. The project can consist of many python(3) files.

A makefile for the specific scenario above could look like this:

PYTHON_FILES := $(shell find src -name '*.py')
TXT_FILES := build/1/test.txt

.PHONY: clean all

all: $(TXT_FILES)

build/1/test.txt: src/1.py
    mkdir -p build/1
    touch build/1/test.txt # emulates: python3 src/1.py
    echo "success!"

clean:
    rm -rf build

Running make with the above project structure and makefile results in the following project structure:

.
├── Makefile
├── build
│   └── 1
│       └── test.txt
└── src
    └── 1.py

How do I generalize the rule head build/1/test.txt: src/1.py to handle projects with any number of python programs (or, equivalently, build subdirectories) and any number of output files per python program?

Gabriel
  • 3
  • 3

1 Answers1

0

You can generalized the existing rule to work on ANY python code in src. Use '%' in the pattern rule, use '$* to refer to the number in the action list.

The rule will re-run the test, whenever the python test is modified. It will record "success" only if the python test indicate completion without an error.

Update 2019-11-24: Generalized the test to handle N tests, each generating multiple files. With rebuild.

Note 1: Make need a way to know if the python test passed without ANY failure. The solution assume non-zero exit code from the python code, or that there is another way to tell if all tests have passed.

Note 2: The done file capture the list of test files generated in the folder (excluding the test.done itself). This can be used to verify that NO output file was removed, if needed, in a separate target to compensate the the lack of explicit files generated by the process

TASK_LIST=1 2 3 4

all: ${TASK_LIST:%=build/%/task.done}

build/%/task.done: src/%.py
        mkdir -p build/$*
        touch build/$*/test.txt # emulates: python3 src/1.py
        # Run script src/%.py should return non-zero exit on failure.
        ls build/$* | grep -xv "$(@F)" > $@
        touch $@ # Mark "success!"

GNU Make documentation: https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html

dash-o
  • 13,723
  • 1
  • 10
  • 37
  • Thanks, however, this does not solve the problem. What happens if `1.py` produces both a `test1.txt` file and a `test2.txt` file? Or, more generally, `n` distinct files? Then you would need `n` distinct make rules too, according to your solution. – Gabriel Nov 23 '19 at 22:18
  • Make will not be able to know how many files SHOULD be created. You can change the rule to create a 'completed' file, which will approximate the behavior that you want - rerun the test, if the PY is modified. I misunderstood the template to imply the test.txt is a a flag file, and NOT a test result file. I'll post extra description. – dash-o Nov 24 '19 at 05:43
  • Solution revised to be more explicit about the role of the test.done as indicator file, and not part of the result set + assumption about being able to know if ALL test passed. – dash-o Nov 24 '19 at 05:56
  • Appreciate your answer, but you misunderstand my question (or I fail to explain my problem). The files are not at all test files: in my application each python program generates multiple plots, matrices and numbers that my LaTeX document then reads in automatically. When stating my problem I tried to generalize it, but perhaps ‘test.txt’ was a bad example name. All I want to do is make exactly one rule to build all output files. I assume this is possible because each output file’s folder determines its python program uniquely. – Gabriel Nov 24 '19 at 09:36
  • I do of course need to explicitly list each output file in some variable in the makefile, but I would like to avoid making more than one rule since the building process (simply running the appropiate python script) is the same. – Gabriel Nov 24 '19 at 09:39
  • Now I'm really confused. I believe that the posted rule will address your requirement - WITHOUT having to list all the output files. You can enumerate the LIST, or even generate it from the list of source files. I've fixed a typo in the 'all' target to make it clear – dash-o Nov 24 '19 at 13:01
  • My bad, I misunderstood your solution: you are right, it solves my problem! Thanks! I like the solution of enumerating the tasks and creating a corresponding `task.done` file to indicate which files where created by the python script. However, is it really necessary with such an extra file? Can't we simply have the build subdirectories as targets in themselves, instead of having each file in each build subdirectory as a target? I should note that I unsuccesfully tried creating a makefile where each file in each build subdirectory is a target in itself, and this led me to posting this question. – Gabriel Nov 24 '19 at 13:32
  • It depends on the robustness of the solution that you want. Consider a case where some a folder content is removed (rm -f build/2/*). If make just test for folder existience, it will not detect that files have disappeared. Also, timestamp on folders are known to challenge Make. I would recommend against using folders are timestamp, and go for the 'done' file. Little noise, but your will usually sleep better. I've seen similar problems when file system were restored, which cause make to get confused. Even the done file approach has limitations, but it will work under normal condition. – dash-o Nov 24 '19 at 14:08
  • I believe you when you say the folders should not be used as targets for the reasons you listed above. However, the current approach also fails if you e.g. call `rm -f build/1/test.txt`. Ideally, Make should detect that the content of `build/1` has changed and rebuild. How would you make this happen? Somehow invoking `make` should check that the content of `build/1/task.done` matches the files in `build/1`. Note that the original (intended) approach of listing each output file explicitly as a target does not suffer from this issue. – Gabriel Nov 24 '19 at 14:36
  • You are correct that unless make knows the full list of generated files, it will not be able to certain problems. But those go beyond the specific question. It's up to you to decide how 'strong' your solution should be. Look at https://stackoverflow.com/questions/4440500/depending-on-directories-in-make, and the referenced question, they will give you some ideas. – dash-o Nov 24 '19 at 15:07