0

Im struggling to get a process started in a Makefile to be launched in the background and the Makefile target to complete.

It's a simple flask and dramatiq application. This is the target:

.PHONY: run-daemon                                                                                                                                                                                                                                             
run-daemon:                                                                                                                                                                                                                                                    
    poetry run gunicorn wsgi:app --daemon --workers 16 --threads 16                                                                                                                                                                                            
    poetry run dramatiq api:broker worker.tasks --processes 1

First I tried using ampersand to background the processes:

.PHONY: run-daemon                                                                                                                                                                                                                                             
run-daemon:                                                                                                                                                                                                                                                    
    poetry run gunicorn wsgi:app --daemon --workers 16 --threads 16 &                                                                                                                                                                                            
    poetry run dramatiq api:broker worker.tasks --processes 1 &

With that gunicorn backgrounds as expected but dramatiq does not.

Then I tried nohup and setsid in various combinations and with and without ampersand but neither of these result in backgrounding dramatiq.

.PHONY: run-daemon                                                                                                                                                                                                                                             
run-daemon:                                                                                                                                                                                                                                                    
    poetry run gunicorn wsgi:app --daemon --workers 16 --threads 16 &                                                                                                                                                                                            
    setsid nohup poetry run dramatiq api:broker worker.tasks --processes 1 &

As it is it does not complete because the dramatiq process does not complete.

But running dramatiq directly like so poetry run dramatiq api:broker worker.tasks --processes 1 results in dramatiq running in the background.

How can I daemonize it such that the make target completes?

joidegn
  • 1,078
  • 9
  • 19
  • 2
    You are looking for a background process, not really a daemon. You can do that with `&` but managing background processes from `make` is quite fragile. Probably rethink your approach, perhaps in terms of a test with a setup fixture to start the required service(s) and a teardown fixture to shut down whatever you started in the startup fixture. – tripleee Apr 11 '22 at 05:57
  • Perhaps see https://stackoverflow.com/questions/44222883/run-a-shell-script-and-immediately-background-it-however-keep-the-ability-to-in for the meaning of the ampersand and some possible complications. – tripleee Apr 11 '22 at 05:59
  • Thanks @tripleee, I know about the ampersand operator and tried it as well but it does not solve the issue. The Make target will not complete with or without the ampersand. I tried setsid and nohup as well with the same effect. – joidegn Apr 11 '22 at 06:17
  • 1
    You probably did something else wrong as well, or `poetry` has something inside it to prevent that from working. Perhaps see also https://stackoverflow.com/questions/37102957/when-writing-a-makefile-how-can-i-kill-a-subprocess-on-exit – tripleee Apr 11 '22 at 06:19
  • Can we know where you tried to put this `&` in your Makefile? – Renaud Pacalet Apr 11 '22 at 14:21
  • Just at the end of the poetry run commands @RenaudPacalet. So `poetry run dramatiq api:broker worker.tasks --processes 1 &` – joidegn Apr 13 '22 at 04:46
  • 2
    And also at the end of the other command? (`poetry run gunicorn wsgi:app --daemon --workers 16 --threads 16 &`). Suggestion: edit your question and show **exactly** what you tried that did not work as you expected. – Renaud Pacalet Apr 13 '22 at 04:59
  • Yeah at the end of both. Thx for the suggestion. Will update. – joidegn Apr 13 '22 at 06:06
  • 1
    The only explanation I see (but you do not give enough information to be sure) is that you do not call `make run-daemon`. You invoke either `make` or `make something` and the corresponding goal is different from `run-daemon`. And it is this other goal that "_does not complete_". – Renaud Pacalet Apr 13 '22 at 07:54
  • Thanks @RenaudPacalet, I have extended the question a bit more as I still have not found the answer. What is weird to me is that I am easily able to background dramatiq when I run the command directly. – joidegn Apr 15 '22 at 12:19
  • @joidegn, you are being too coy. Although we do appreciate keeping the question focused, you need to give us enough specifics to work with. In this case, a [mre]. That would have the form of a complete makefile with which the issue can be reproduced, the `make` command with which to produce it, and any required inputs. Note also that `poetry` itself is probably not relevant to the issue, and very likely you can change it to any dummy command that runs a longish time, such as `sleep 30`, for demo purposes (and if not, then that's a valuable piece of information). – John Bollinger Apr 15 '22 at 13:10
  • Another possibility is that one of these commands you want to run in the background waits for an input and gets stuck because the input never comes... – Renaud Pacalet Apr 15 '22 at 13:13

1 Answers1

1

You need to use backslash to create multiline combos executed as a single shell command.

I would say this should do (pay attention to escaping $ and Co.):

SHELL:=/bin/bash
.PHONY run-daemon
run-daemon:
        daemon-executable > daemon.log 2>&1&\
        disown -h $$!
bobah
  • 18,364
  • 2
  • 37
  • 70
  • This does not seem to be quite right because I get `disown -h $? /bin/sh: line 2: disown: 0: no such job make: *** [Makefile:35: run-daemon] Error 1` but interestingly the redirection from stderr to stdout seems to make the ampersand work correctly. I have no idea why but your answer works for me even without the disown command. Fwiw running echo `$?` seems to be `0` if I run `daemon-executable > daemon.log 2>&1&`directly which makes sense to me. Was $? supposed to contain the process PID? – joidegn Apr 15 '22 at 12:28
  • I also was not intending to do multiline combos at all, merely to run two commands each of which should result in a backgrounded process. – joidegn Apr 15 '22 at 12:29
  • Feel free to adjust the response so that I can run it with the disown command and I'll mark it as the correct answer! – joidegn Apr 15 '22 at 12:51
  • `$?` was supposed to be the last PID, yes. And `disown -h` is supposed to be a bash built-in to disown a process instead of `nohup`. Let me try it in a shell. – bobah Apr 15 '22 at 12:52
  • It should be `$!`, not `$?`, and it also requires to switch shell to be `bash` with `SHELL:=/bin/bash` (not sure it is a universally good advise, if you need portability). – bobah Apr 15 '22 at 13:13