0

I want to execute a respawning bash script which looks like the following

until python3 myapp.py; do
  echo "$(date) -- every exeception" | tee -a /var/log/app.log
  sleep 1;
done

This is basically respawning the script if it dies.

In the example of the link it only logs the $? which might just be a number. But I wish to tee the exceptions that occur when the script fails to the log/app.log file.

I tried different stuff but nothing works. Any direction how I can get the exceptions logged in the above mentioned way.

trial

until python3 myapp.py; do
    echo "$(date) > &2 >&1 | tee -a /var/log/app.log
    sleep 1;
done

Output from log file

Fri May 18 18:09:03 UTC 2018 > 2>&1
Fri May 18 18:09:03 UTC 2018 > 2>&1
Fri May 18 18:09:04 UTC 2018 > 2>&1
Fri May 18 18:09:05 UTC 2018 > 2>&1
Fri May 18 18:09:06 UTC 2018 > 2>&1
Fri May 18 18:09:06 UTC 2018 > 2>&1
Fri May 18 18:09:07 UTC 2018 > 2>&1
Fri May 18 18:09:29 UTC 2018 >&2>&1
Fri May 18 18:09:30 UTC 2018 >&2>&1
Fri May 18 18:09:30 UTC 2018 >&2>&1
Fri May 18 18:09:31 UTC 2018 >&2>&1
Fri May 18 18:09:32 UTC 2018 >&2>&1
Fri May 18 18:09:33 UTC 2018 >&2>&1
Shan-Desai
  • 3,101
  • 3
  • 46
  • 89
  • 1
    Put the `| tee -a ...` after the `done`. You might need `done 2>&1 | tee -a` if the exception is written to stderr. – Charles Duffy May 18 '18 at 18:21
  • ...right now, you're only `tee`ing the `echo`, not the `python3 myapp.py` call. – Charles Duffy May 18 '18 at 18:22
  • Another option is `until python3 myapp.py > >(tee -a /var/log/app.log) 2>&1; do ...` -- the point of using a process substitution instead of a regular pipeline there is to avoid disrupting the exit status (as unless the `pipefail` runtime flag is set, the exit status of a pipeline is that of its last component). – Charles Duffy May 18 '18 at 18:22
  • I tried `until python3 app.py; do sleep 1; done 2>&1 | tee -a /var/log/app.log` but the script didn't respawn – Shan-Desai May 18 '18 at 19:14
  • Can you provide a reproducer for that? If I run `stub() ( sleep 3; exit 1; ); until stub; do sleep 1; echo "respawning"; done 2>&1 | tee -a stub-test.log`, it respawns just fine. An issue we can't see ourselves without your `app.py` is an issue nobody can check their answers for. – Charles Duffy May 18 '18 at 20:00
  • (BTW, what's your operating system? The better practice is to use whatever process supervision system is supported by your OS vendor -- systemd, upstart, launchd, etc -- to do the respawning; when that's done correctly, you have a chain of processes that each check their childrens' status and recover from failures all the way back to PID 1). – Charles Duffy May 18 '18 at 20:04
  • I actually am using a custom yocto image with `linux-mainline` kernel on a board for embedded application. Sadly there is no `cron` available but I can perhaps create a `systemd` script. If the code crashes will `systemd` restart it? – Shan-Desai May 18 '18 at 20:06
  • I'm not familiar with Yocto -- if your OS vendor doesn't already set up/support systemd, it's probably better to use a simpler init system -- personally, I'm very fond of [Runit](http://smarden.org/runit/). Either *will* indeed restart your service for you (if configured to do so); see http://smarden.org/runit/runscripts.html to see just how simple a runit script is. – Charles Duffy May 18 '18 at 20:21
  • @CharlesDuffy went ahead and changed the whole outlook of the problem with a long but working solution. Care having a look? – Shan-Desai May 20 '18 at 15:44

1 Answers1

0

I ended up writing a logging handler for the application I have and decided to use the until loop into a tmux session. The Session is created using a systemd service (hinted by @CharlesDuffy)

Session creator bash script:

#!/bin/bash
sleep 1;

/usr/bin/tmux new-session -d -s test
/usr/bin/tmux set-option set-remain-on-exit on

/usr/bin/tmux new-window -d -n 'testpy' -t test:1 'sleep 1; until usr/bin/python3 /home/root/test/test.py; do echo $(date) 2>&1; sleep 1; done'

gave it executable rightschmod +x testsess

Created a systemd script in the /etc/systemd/system/multi-user.wants/testpy.service

testpy.service

[Unit]
Description=test service
After=influxdb.service

[Service]
Type=simple
ExecStart=/home/root/test/testsess
ExecStop=/usr/bin/tmux kill-session -t test
RemainAfterExit=true
Restart=on-failure

[Install]
WantedBy=multi-user.target

(not a systemd champ here! but the script works)

reloaded the daemon

systemctl daemon-reload

and started the service

systemctl start testpy.service

Checks

tmux ls

should create a test session and in window nr. 1 the script respawns even on failure.

Shan-Desai
  • 3,101
  • 3
  • 46
  • 89
  • 1
    Personally, I would have skipped the use of tmux here -- it provides an indirection layer that makes systemd's ability to tell if the process under supervision is active less accurate, and prevents output from being logged through journald as is default for other services -- but this does generally look workable, even so. – Charles Duffy May 20 '18 at 19:28