14

One target in my makefile is a very CPU and time consuming task. But I can split the workload and run the task several times in parallel to speed up the entire process.

My problem is that make doesn't wait for all processes to complete.

Consider this simple script, named myTask.sh:

#!/bin/bash

echo "Sleeping $1 seconds"
sleep $1
echo "$1 are over!"

Now, let's call this from a bash script, and use wait to wait for all tasks to complete:

#!/bin/bash

echo "START"
./myTask.sh 5 &
./myTask.sh 15 &
./myTask.sh 10 &

wait  # Wait for all tasks to complete

echo "DONE"

The output is as expected:

START
Sleeping 15 seconds
Sleeping 5 seconds
Sleeping 10 seconds
5 are over!
10 are over!
15 are over!
DONE

But when trying the same in a Makefile:

test:
    echo "START"
    ./myTask.sh 5 &
    ./myTask.sh 15 &
    ./myTask.sh 10 &
    wait
    echo "DONE"

it doesn't work:

START
Sleeping 5 seconds
Sleeping 15 seconds
Sleeping 10 seconds
DONE
sweber@pc:~/testwait $5 are over!
10 are over!
15 are over!

Of course, I could create multiple targets which can be "built" in parallel by make, or let make just run the bash script which runs the tasks. But is there a way to do it more like what I already tried?

sweber
  • 2,916
  • 2
  • 15
  • 22
  • My first attempt would be to put all relevant commands in the same line: `./myTask.sh 5 & ./myTask.sh 15 & ./myTask.sh 10 & wait` – melpomene Jan 03 '17 at 21:32
  • 2
    Ideal job for **GNU Parallel**... `parallel ./myTask.sh ::: 5 10 15` – Mark Setchell Jan 03 '17 at 22:00
  • If you are using **GNU make**, and the `ONESHELL` feature is installed, your script will work *"as-is"* if you add a line with `.ONESHELL:` at the start since that will force your entire recipe to run in a single invocation of your shell. – Mark Setchell Jan 06 '17 at 17:02

2 Answers2

16

Why not use the mechanisms already built in make? Something like this:

task%:
        sh task.sh $*

test: task5 task15 task10
        echo "DONE"

You will be able to adjust the level of parallelism on the make command line, instead of having it hard-coded into your makefile (e.g use 'make -j 2 test' if you have 2 cores available and 'make -j 32 test' if you have 32 cores)

Come Raczy
  • 1,590
  • 17
  • 26
  • 1
    How do you attach parameters please? E.g. `task 5` rather than `task5`. – Mark Setchell Jan 04 '17 at 00:47
  • 1
    @MarkSetchell what do you mean? task5 is the name of a target, it has to meet the syntactic requirements for a target. In the recipe for the `task5` target, the stem $* will be 5 and you can do whatever you want with that (e.g. use it as a parameter for the `task.sh` script as shown in the example above) – Come Raczy Jan 04 '17 at 00:53
  • 1
    You can automate the level of parallelism with `-j $(nproc)` instead of using an integer constant. `nproc` outputs the number of processors. – doug65536 Aug 09 '18 at 20:39
15

Each individual logical line in a recipe is invoked in its own shell. So your makefile is basically running this:

test:
        /bin/sh -c 'echo "START"'
        /bin/sh -c './myTask.sh 5 &'
        /bin/sh -c './myTask.sh 15 &'
        /bin/sh -c './myTask.sh 10 &'
        /bin/sh -c 'wait'
        /bin/sh -c 'echo "DONE"'

You should make it all run in a single shell using semicolons and backslash/newlines to combine the physical lines into one logical line:

test:
        echo "START"; \
        ./myTask.sh 5 & \
        ./myTask.sh 15 & \
        ./myTask.sh 10 & \
        wait; \
        echo "DONE"
MadScientist
  • 92,819
  • 9
  • 109
  • 136
  • `&` is already a command separator. You don't need an extra `;`. – melpomene Jan 03 '17 at 21:37
  • _Each individual logical line in a recipe is invoked in its own shell._ That was the missing point. Thanks! – sweber Jan 04 '17 at 14:04
  • Is it possible to stop this task with Cntrl+C? It "kills" the makefile, but the task keeps running for me – Lucas Bustamante Sep 09 '20 at 19:33
  • You can but you'll have to do it yourself with a more complex script. There's nothing in make itself which will do this, because make has no idea what other processes are running so it can't kill them. You can write your script to use the shell's `trap` function to catch an interrupt signal and kill all the job(s) that are running in the background. – MadScientist Sep 09 '20 at 21:17