How to log errors in python when using the os module

Question

I'm trying to incorporate a simple way to keep track of a periodic mysqldump command I want to run using the os module in python. I've written this, but in testing it doesn't raise the exception, even when the mysqldump command completes with an error. I'm pretty new to python, so I might be approaching this terribly, but I thought I would try to get pointed in the right direction.

db_dump = "mysqldump -u %s -p%s --socket=source_socket --databases %s | mysql -u %s -p%s   --socket=dest_socket" % (db_user, db_pass, ' '.join(db_list), db_user, db_pass)

try:
    os.system(db_dump)
except:
    logging.error("databases did not dump")
else:    
    logging.info("database dump complete")

write `logging`s output to a file [Using the logging python class to write to a file?](http://stackoverflow.com/questions/6386698/using-the-logging-python-class-to-write-to-a-file) — , Dec 06 '12 at 22:12

score 5 · Answer 1 · answered Dec 06 '12 at 22:12

5

os.system is not a very robust or powerful way to call system commands, I'd recommend using subprocess.check_output() or subprocess.check_call

ie,

>>> cmd = 'ls -l'
>>> badcmd = 'ls /foobar'
>>> subprocess.check_call(cmd.split())
0
>>> subprocess.check_call(badcmd.split())
ls: /foobar: No such file or directory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ls', '/foobar']' returned non-zero exit status 1

answered Dec 06 '12 at 22:12

Cameron Sparr

3,925
2
22
31

Thank you for the tip. I will research subprocess and try to convert my code to use it once I'm comfortable with it. – numb3rs1x Dec 06 '12 at 22:25
+1, but you shouldn't do `cmd.split()`. Just pass `cmd` as a string, and `subprocess` will break it up in a platform-appropriate way, taking into account quotes. (Of course it's better to have a list of args than a string in the first place—but if you do have a string, do not call `split` just to get a `list` of `args`.) – abarnert Dec 06 '12 at 22:27
@numb3rs1x: It really doesn't take much to convert your code—just replace every use of `os.system` with `subprocess.check_call`. You could even just search-and-replace this. It won't exactly the same thing `os.system` did—instead, it will do *what you expected `os.system` to do*, which is better. (If you _do_ need anything fancy that `os.system` did, like passing shell expressions in the command, see the `shell=True` flag. But you probably don't.) – abarnert Dec 06 '12 at 22:29
@abamert: I believe you need to `split()` the command unless you call it with `shell=True` – Cameron Sparr Dec 06 '12 at 22:47
@abamert I've been playing around with the subprocess module, and it does seem to be the way to go for running a shell. I'm struggling with how to handle the exceptions is raises so they go into a log file of my choosing. – numb3rs1x Dec 07 '12 at 06:03

steveha · Accepted Answer · 2012-12-06T22:18:07.253

2

os.system() returns an integer result code. When it returns 0, the command ran successfully; when it returns a nonzero value, that indicates an error.

db_dump = "mysqldump -u %s -p%s --socket=source_socket --databases %s | mysql -u %s -p%s   --socket=dest_socket" % (db_user, db_pass, ' '.join(db_list), db_user, db_pass)

result = os.system(db_dump)
if 0 == result:
    logging.info("database dump complete")
else:
    logging.error("databases did not dump; result code: %d" % result)

Like @COpython, I recommend the use of subprocess. It is a bit more complicated than os.system() but it is tremendously more flexible. With os.system() the output is sent to the terminal, but with subprocess you can collect the output so you can search it for error messages or whatever. Or you can just discard the output.

edited Dec 06 '12 at 22:18

answered Dec 06 '12 at 22:11

steveha

74,789
21
92
117

Thank you steveha for the quick and friendly response. I tried this and it worked, but I will research subprocess to see if that would be a better solution ultimately for this code. – numb3rs1x Dec 06 '12 at 22:28
With `os.system()` you get the exit status from the program you run. You can look at the "man page" for the program to see what error it returns. With `subprocess` you can get the exit status as well, but with `os.system()` that's all you get; you don't get an exception. – steveha Dec 06 '12 at 22:41

score 2 · Answer 3 · answered Dec 06 '12 at 22:27

2

Here is what I would do.

import logging
import subprocess
log = logging.getLogger(__name__)

cmd = "mysqldump -u %s -p%s --socket=source_socket --databases %s | mysql -u %s -p%s " \
      "--socket=dest_socket" % (db_user, db_pass, ' '.join(db_list), db_user, db_pass)

process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE.PIPE)
stdout, stderr = process.communicate()
stdout = [x for x in stdout.split("\n") if x != ""]
stderr = [x for x in stderr.split("\n") if x != ""]

if process.returncode < 0 or len(stderr):
    for error in stderr:
        log.error(error)

answered Dec 06 '12 at 22:27

rh0dium

6,811
4
46
79

This looks a lot more advanced than I'm prepared for at my novice python level. I do thank you for contributing, and I'm looking forward to the day I when I have a better understanding of the intricacies of python programming. – numb3rs1x Dec 06 '12 at 22:32
Actually this isn't so bad. All we do is shove your command into subprocess (which is a better os.system - see http://stackoverflow.com/questions/4813238/difference-between-subprocess-popen-and-os-system). Once we do that we wait for the results (communicate). Then split up the resulting lines into lines again, verify we got a good return and if not dump the output. – rh0dium Dec 06 '12 at 22:37

How to log errors in python when using the os module

3 Answers3