variable expansion in subshell in makefile fails

Question

I have a makefile function inside a makefile (myfunction.mk):

.ONESHELL:

define call_script
set +x
mkdir -p $$(dirname $(2))
if [ ! -f $(2) ]; then
echo "" > $(2)
fi
REDIRECT='| tee -a'
echo '>> $(1)'
($(1) ???????? $(2))
RET_CODE=$$?
echo "exit_code is: $$RET_CODE"
if [ ! $$RET_CODE = 0 ]; then 
echo "$(3) terminated with error $$RET_CODE"
exit $$RET_CODE
else 
if [ ! -z "$(strip $(3))" ]; then
echo "$(3) done"
fi
fi
endef

this function call a script and append result to a log (which is created with its folder if non existing), the result of the script is append only if the makefile variable given as the 4th ($(4)) argument is equal to 'yes'.

you call it like this:

include myfunction.mk

OUTPUT_ENABLED ?= yes

target:
  $(call call_script, echo "test", reports/mylog.log, "doing test", OUTPUT_ENABLED)

This works for the most part:

if i replace '????????' by '| tee -a', it works.
if i replace '????????' by $(REDIRECT), it fails.
if i replace '????????' by $$REDIRECT, it fails.

why?

note: running it from a shell /bin/sh: symbolic link to dash

note: of course i want to add a ifeq that allows me to check for $(4) and replace | tee -a by &>>

What you show is a bit strange. No target at all? You invoke `call` like this? I do not understand. — Renaud Pacalet, Dec 02 '20 at 15:16
addtionnal information added for your understanding with separate files — Guillaume D, Dec 02 '20 at 15:57
To add to the (great) answer below, the reason you can't use `$(REDIRECT)` or use `ifeq` etc. is that those are _make_ variables etc. but you're running this in the shell. When you have `REDIRECT=xxx` in your shell script that sets the _shell_ variable `REDIRECT`, not the _make_ variable `REDIRECT`, so you can't use make constructs with it. — MadScientist, Dec 02 '20 at 16:26
One more remark: if you have one separate Makefile (`myfunction.mk`) for your make macro, it would probably make more sense to have a `myfunction.sh` shell script, instead, and call the script in the recipe of your main Makefile. The script would be less clumsy, easier to maintain, less error prone. Make is a beautiful tool but when the recipes become too complicated, even if you hide this with macros, it is usually the sign that other more suitable tools should be involved. — Renaud Pacalet, Dec 03 '20 at 07:32
the issue with calling a shell function using a shell lib, is that i have for every line (because make opens a shell for every line) or for every target (using .ONESHELL) , to type ```source myscript.sh```. is there a way to avoid that? — Guillaume D, Dec 03 '20 at 08:50
https://stackoverflow.com/questions/7507810/how-to-source-a-script-in-a-makefile — Guillaume D, Dec 03 '20 at 09:04

Renaud Pacalet · Accepted Answer · 2020-12-02T16:19:18.760

I'll assume that you use call in a recipe, not flat in your Makefile. There are few problems with your shell script. First, if you try the following on the command line:

mkdir -p reports
REDIRECT='| tee -a'
echo '>> echo "test"'
(echo "test" $REDIRECT reports/mylog.log)

you'll see that echo considers:

"test" $REDIRECT reports/mylog.log

as its arguments. They are expanded and echoed, which prints:

test | tee -a reports/mylog.log

on the standard output, not the effect you expected, I guess. You could, for instance, use eval. On the command line:

eval "echo "test" $REDIRECT reports/mylog.log"

Which, in your Makefile, would become:

eval "$(1) $$REDIRECT $(2)"

Next you should not quote the third parameter of call because the quotes will be passed unmodified and your script will be expanded by make as:

echo " "doing test" terminated with error $RET_CODE"

Again probably not what you want.

Third, you should avoid useless spaces in the parameters of call because they are preserved too (as you can see above between the first 2 double quotes):

.PHONY: foo
foo:
    $(call call_script,echo "test",reports/mylog.log,doing test,OUTPUT_ENABLED)

And for your last desired feature, it would be slightly easier to pass the value of OUTPUT_ENABLED to call instead of its name, but let's go this way:

$ cat myfunction.mk
define call_script
set +x
mkdir -p $$(dirname $(2))
if [ ! -f $(2) ]; then
echo "" > $(2)
fi
if [ "$($(4))" = "yes" ]; then
REDIRECT='| tee -a'
else
REDIRECT='&>>'
fi
echo '>> $(1)'
eval "$(1) $$REDIRECT $(2)"
RET_CODE=$$?
echo "exit_code is: $$RET_CODE"
if [ ! $$RET_CODE = 0 ]; then 
echo "$(3) terminated with error $$RET_CODE"
exit $$RET_CODE
else 
if [ ! -z "$(strip $(3))" ]; then
echo "$(3) done"
fi
fi
endef
$ cat Makefile
.ONESHELL:

include myfunction.mk

OUTPUT_ENABLED ?= yes

target:
    $(call call_script,echo "test",reports/mylog.log,doing test,OUTPUT_ENABLED)

Note that I moved the .ONESHELL: in the main Makefile because it is probably better to not hide it inside an included file. Up to you.

if i want to do a "$$?" after the "$(1)" call, how can i do it with the eval?? if I put it after the eval i guess it would take the eval return code? thank you — Guillaume D, Dec 02 '20 at 16:03
The `eval` return code is the return code of the command it evaluated. — Renaud Pacalet, Dec 02 '20 at 16:13
Tangentially, notice also that `cmd; if [ $? != 0 ]` is just an unidiomatic way to say `if ! cmd`; see also [Why is testing “$?” to see if a command succeeded or not, an anti-pattern?](https://stackoverflow.com/questions/36313216/why-is-testing-to-see-if-a-command-succeeded-or-not-an-anti-pattern) — tripleee, Dec 02 '20 at 16:25
your answer is almost what i am looking for, except that if i do ```eval "$(1) $$REDIRECT $(2)"``` it does not stop on error, but the line ```$(1)``` does. Furthermore, ```$(3) done``` is displayed before ```$(1)``` is finished... not what i want — Guillaume D, Dec 03 '20 at 08:29
The most problematic issue here is that if you pipe your commands, the exit code is the exit code of the last command in a pipe, e.g `false | tee foo.log` will exit with 0 as `tee` will most probably succeed. Note also that pipe only redirects stdout, so your log will not contain any stderr messages, unless called as `2>&1 | tee -a`. — raspy, Dec 03 '20 at 12:02

score 0 · Answer 2 · answered Dec 03 '20 at 22:45

The most problematic issue here is that if you pipe your commands, the exit code is the exit code of the last command in a pipe, e.g false | tee foo.log will exit with 0 as tee will most probably succeed. Note also that pipe only redirects stdout, so your log will not contain any stderr messages unless explicitly redirected.

Considering that piping commands influence exit code and lack of portability of $PIPESTATUS (most specifically not being supported in dash), I would try to avoid piping commands and use a temporary file for gathering output, i.e.:

$ cat Makefile
# $(1) - script to execute
# $(2) - log file
# $(3) - description
define call_script
echo '>> $(1)'
$(if $(OUTPUT_ENABLED), \
  $(1) > $@.log 2>&1; RET_CODE=$$?; mkdir -p $(dir $(2)); cat $@.log >> $(2); cat $@.log; rm -f $@.log, \
  $(1); RET_CODE=$$? \
); \
echo "EXIT_CODE is: $${RET_CODE}"; \
if [ $${RET_CODE} -ne 0 ]; then $(if $(3),echo "$(3) terminated with error $${RET_CODE}";) exit $${RET_CODE}; fi; \
$(if $(3), echo "$(3) done.")
endef

good:
        $(call call_script,echo "test",reports/mylog.log,doing test)

bad:
        $(call call_script,mkdir /root/foo,reports/mylog.log,intentional fail)

ugly:
        $(call call_script,bad_command,reports/mylog.log)

Regular call will not create the logs and will stop on errors:

$ make good bad ugly
echo '>> echo "test"'
>> echo "test"
echo "test"; RET_CODE=$? ; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "doing test terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi;  echo "doing test done."
test
EXIT_CODE is: 0
doing test done.
echo '>> mkdir /root/foo'
>> mkdir /root/foo
mkdir /root/foo; RET_CODE=$? ; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "intentional fail terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi;  echo "intentional fail done."
mkdir: cannot create directory ‘/root/foo’: Permission denied
EXIT_CODE is: 1
intentional fail terminated with error 1
make: *** [Makefile:19: bad] Error 1

Note that ugly was not built due to failure on bad. Now the same with the log:

$ make good bad ugly OUTPUT_ENABLED=1
echo '>> echo "test"'
>> echo "test"
echo "test" > good.log 2>&1; RET_CODE=$?; mkdir -p reports/; cat good.log >> reports/mylog.log; cat good.log; rm -f good.log; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "doing test terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi;  echo "doing test done."
test
EXIT_CODE is: 0
doing test done.
echo '>> mkdir /root/foo'
>> mkdir /root/foo
mkdir /root/foo > bad.log 2>&1; RET_CODE=$?; mkdir -p reports/; cat bad.log >> reports/mylog.log; cat bad.log; rm -f bad.log; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "intentional fail terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi;  echo "intentional fail done."
mkdir: cannot create directory ‘/root/foo’: Permission denied
EXIT_CODE is: 1
intentional fail terminated with error 1
make: *** [Makefile:19: bad] Error 1

$ cat reports/mylog.log
test
mkdir: cannot create directory ‘/root/foo’: Permission denied

Note that this time ugly was also not run. But if run later, it will correctly append to the log:

$ make ugly OUTPUT_ENABLED=1
echo '>> bad_command'
>> bad_command
bad_command > ugly.log 2>&1; RET_CODE=$?; mkdir -p reports/; cat ugly.log >> reports/mylog.log; cat ugly.log; rm -f ugly.log; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then  exit ${RET_CODE}; fi;
/bin/sh: 1: bad_command: not found
EXIT_CODE is: 127
make: *** [Makefile:22: ugly] Error 127

$ cat reports/mylog.log
test
mkdir: cannot create directory ‘/root/foo’: Permission denied
/bin/sh: 1: bad_command: not found

Personally I am not fan of implementing logging in this way. It is complicated and it only logs output of commands, not make output itself, and only of those commands which are explicitly called to do so. I'd rather keep Makefile clean and simple and just run make 2>&1 | tee log instead to have the output logged.

variable expansion in subshell in makefile fails

2 Answers2