Prevent Python script to fail

Question

I have a python script that will be running 24/7 on a raspberry for at least five month, without a way for me to access the thing to reboot it if anything goes wrong. It's using various libraries to connect to a sql database, wireless RFID readers etc.

It's basically a few dozen lines of defining functions, connecting the raspberry to the devices via usb/serial, and then a big while True: loop.

I want to make sure that if an error happens, the code will keep going (or try again until no error occurs) and not get stuck on the error. I've read that I can use the try: Except: but it seems to be not be recommended.

Do you have any ressources where I can learn how to make a python script bulletproof whatever happens, or ideas about how to do that ?

It really depends on what kind of failure you expect (and want to handle), but generally I would say you should leave that "keep alive" / automated restarted / ... out of your python code and instead have your distro take care of it. Start the process and keep doing so if it terminated. Depending on your init and its abilities, you may already get that functionality and just register another service. That could also cover the host being power cycled which you really cannot do anything for within your script itself. — Ondrej K., Sep 14 '19 at 16:46

score 1 · Answer 1 · answered Sep 14 '19 at 17:04

try is a great resource.

Here, I attempt to call an environment variable. If it is not set, I attempt to set it. If I cannot set it I let the user know what is going on so they don't get scared.

try:
    os.environ['R_HOME']
except:
    print("""
        \u2717 ERROR: The default Jupyter/Conda path for R, `os.environ['R_HOME']` is undefined, cannot print it's path.
        This should have been set when running the Docker image like so:
        `-e R_HOME=/opt/conda/lib/R`
    """)
    try:
        print("\u0009Attempting to set `os.environ['R_HOME']` manually...")
        os.environ['R_HOME'] = '/opt/conda/lib/R'
    except:
        print("\u0009Failed to set `R_HOME` this must not be a conda managed jupyter environment with both R and Python kernels.")
    else:
        print("\u0009\u2713 -- 'R_HOME' is now `" + os.environ['R_HOME'] + "`." ) 
else:
    print("\u0009\u2713 -- The path of the Jupyter R enviroment being accessed by `rpy2` is '" + os.environ['R_HOME'] + "'.\n")

JL Peyret · Answer 2 · 2019-09-15T16:02:37.320

register it with systemd, monit or runit. these are service managers who, when the programs they are watching die, note its absence and restart it with the command line mechanism you’ve provided. might be a bit tricky to get your head around them (and provide the pid or other scheme to “sign” a python script’s presence) but it’s what they’re designed to do. I am quite fond of runit myself but systemd is (perhaps unfortunately) present on most Linux distributions.

other stuff you rely on, not just your script, may also need similar overwatch, though typically things like database servers already configure themselves to do this from the start.

Past ensuring good coding hygiene, quality and testing, addressing your type of requirement in any program itself (in Python or other languages) is an unproductive and falsely reassuring pursuit. High availability mechanisms rely on worker/master or sibling systems where a second program restarts the first. Not on “hoping for the best” from your own programming skills.

This, as @Ondrej K also said, is best left to your distro - init is another framework dedicated to this. Five months of uptime is a totally reasonable expectation from a well-configured Linux system. And, as he pointed out, these mechanisms also cover system reboots.

Just to be clear: you still need to ensure your program can handle itself under all circumstances - cycling restarts won’t help if it’s stuck on a write permission error for example.

P.S. if the Pi’s networked, ssh might be helpful if you need to access it.

score 0 · Answer 3 · answered Sep 14 '19 at 18:18

Actually, the try: except: block should be fine, with the caveat that you should be sure to log the error and stack trace in your except: block; in your case, this could simply mean either printing to a console that saves its output, saving the logs to a log file, and/or having your script send you an e-mail. For example:

import traceback

try:
    #Code that may cause an exception.
except Exception as e:
    print(e)
    print(traceback.format_exec())

Prevent Python script to fail

3 Answers3