write to file is empty when I write from within a docker img

Question

The code below is some step that is part of a MakeFile which is executed when a certain stage is run in ci/cd.

.PHONY:deploy_endpoint_configuration
deploy_endpoint_configuration: 
    @echo Deploying endpoint configuration
    gcloud endpoints services deploy ocr/model_adapter/predict_contracts.yaml --project $(project) &> $(project_dir)/deploy_contract.txt
    cat $(project_dir)/deploy_contract.txt | grep -Eo '\d*-\d*-\w*' | tail -n1 > $(project_dir)/config_id.txt

What I'm experiencing is that the content of deploy_contract.txt etc is always empty when this piece of code is executed in the pipeline (e.g. GitLab). I don't understand why? Has this to do w/ the fact that MakeFile executes a new shell for every command? Not entirely sure though, and yet this is hard to debug. I do confirm this issue when I run it as followed: gitlab-runner exec docker -here the stage- (for local debug purpose). But when I run it locally on macOS (i.e. only execute make deploy_endpoint_configuration), thus it's not wrapped in a container as before, it runs and functions like it should (read as the content of config_id and deploy_contract is not empty but contains the stdout + errorout)

for reference: image used in the ci/cd stage = image: dsl.company.com:5000/python:3.7-buster on top of that gcloud cli is installed to make use of gcloud commands.

Anyone an idea to why no content is written to my files? (it's for sure deploying though - so there must be something)

Can you confirm that `gcloud endpoints services deploy ocr/model_adapter/predict_contracts.yaml --project $(project)` is outputting something? — tankthinks, Apr 12 '22 at 14:42
I would try running the container in interactive mode, `-it`, then running `make` manually inside the running container. You can try `make -n ` to see what commands make would have run. — tankthinks, Apr 12 '22 at 14:43
@tankthinks indeed it's outputting relevant data. I'll try your suggestion — EiJay, Apr 12 '22 at 15:08

score 0 · Accepted Answer · answered Apr 12 '22 at 19:10

My suspicion is that it's because your command script relies on bash features. GNU make will always run /bin/sh as its shell. On some systems (like RedHat and RedHat-derived systems) /bin/sh is actually a link to bash and so makefile recipes that use bash-specific features will work. I believe MacOS does the same, although I don't do Mac.

On other systems, like Debian and Debian-derived systems like Ubuntu, /bin/sh is a simple, fast POSIX-compliant shell like dash, which doesn't support fancy bash things and so makefile recipes that use bash-specific features will not work.

Probably your container image is one of the latter type, while MacOS (your local system) uses bash as the shell.

You are using &> which is a bash-specific feature, that is not supported by POSIX shells.

You should write this using POSIX syntax, which is >$(project_dir)/deploy_contract.txt 2>&1

Either that or you can try overriding the shell in your makefile:

SHELL := /bin/bash

to force make to run bash instead... this will work as long as the container image you're running does actually have bash in it.

thanks a lot for the help. I was able to reshape my understanding based on your comment "You should write this using POSIX syntax, which is >$(project_dir)/deploy_contract.txt 2>&1" in addition, the following was very helpful on top of your feedback to this matter: https://stackoverflow.com/questions/818255/in-the-shell-what-does-21-mean#:~:text=%40dbr%20cmd%202,as%20they%20occur. — EiJay, Apr 13 '22 at 09:31

write to file is empty when I write from within a docker img

1 Answers1