How to log error to file, and not fail on exception

Question

I am downloading a file from the net, and it fails even though I am doing:

for p in query:

    try:
    
    except IOError as e:
       print e;

If there is an error, I want to log it, and then continue on with the next file.

In this loop, I am trying to download an image, if for some reason the filename was bad, or the website was down, etc., I want to continue with the next item in the for loop.

Is there a more generic error that won't fail and continue processing?

Also, how can I log errors to a file?

Once the error occurs, you cannot continue -- the download is broken. What specific error are you getting? — S.Lott, Aug 01 '10 at 21:54
I have this in a for loop, why can't I catch the error, then continue and try the next file? — Blankman, Aug 01 '10 at 21:55
@Blankman: Since you didn't provide any code, it's impossible to guess why you can't "catch the error, then continue and try the next file". — S.Lott, Aug 01 '10 at 22:00
@Blankman: Since you still didn't provide enough code, we still can't determine why your loop doesn't work. — S.Lott, Aug 01 '10 at 22:34
@Blankman: What is the error message (traceback) you get when the script stops? Is it an `urllib2.HTTPError` for instance? At the moment it seems you may only be catching `IOError`. — unutbu, Aug 01 '10 at 22:38

score 53 · Answer 1 · answered Aug 01 '10 at 21:51

53

You could use the logging module:

import logging
logging.basicConfig(filename='/tmp/myapp.log', level=logging.DEBUG, 
                    format='%(asctime)s %(levelname)s %(name)s %(message)s')
logger=logging.getLogger(__name__)

try:
    1/0
except ZeroDivisionError as err:
    logger.error(err)

Running the script writes in /tmp/myapp.log:

% cat /tmp/myapp.log 
2010-08-01 17:50:45,960 ERROR __main__ integer division or modulo by zero

answered Aug 01 '10 at 21:51

unutbu

842,883
184
1,785
1,677

2

Nice. But the download is still broken. This won't magically make it "[not] fail and continue processing". – S.Lott Aug 01 '10 at 21:55
Why do I have to manually write the error to the log file. In batch script you could set it up such that all errors automatically get written to the Log file. You do that by putting your error prone code or your entire code within brackets and redirecting the error stream to a log file. Is there no way to implement this in Python, such that all errors automatically get written in a log file. – Ghos3t Jul 24 '18 at 04:50
1

@Ghos3t: You might be looking for how to [redirect stderr](https://docs.python.org/3/library/contextlib.html#contextlib.redirect_stdout) or how to [log uncaught exceptions](https://stackoverflow.com/q/6234405/190597). If that is not what you are looking for, please ask a new question with all the details. – unutbu Jul 24 '18 at 14:42

score 36 · Accepted Answer · answered Aug 01 '10 at 22:09

As pointed out by Lott, if a download is failing, unless the problem is fixed upstream (or with your download address), the best you can do is to try again. However, if the situation is that you have a list of downloads, and simply want to skip over failed downloads instead of exiting, then:

logf = open("download.log", "w")
for download in download_list:
    try:
        # code to process download here
    except Exception as e:     # most generic exception you can catch
        logf.write("Failed to download {0}: {1}\n".format(str(download), str(e)))
        # optional: delete local version of failed download
    finally:
        # optional clean up code
        pass

Things to note:

(1) Use of the "logging" module, as suggested by ~unutbu, gives you much more flexibility and power with your logging output, including time stamp, simultaneously writing to different channels (e.g., stderr, file) depending on error levels, etc. etc.

(2) You might consider implementing the above logic using the "with" construct.

what is logf, can't find it anywhere when searching for 'python logf' in google. — Blankman, Aug 01 '10 at 22:20
Thanks Jeet. It worked for me, but after adding logf.close(). Closing the file is important. — Shalini Baranwal, Sep 24 '19 at 08:25
it does what I want it to do, thanks! But how would I continue the code after the error is logged? — Dekriel, Mar 11 '22 at 13:59

score 20 · Answer 3 · edited Jul 07 '20 at 05:32

20

This will write your error to a log file and continue to run the code.

import traceback

#This line opens a log file
with open("log.txt", "w") as log:

    try:
        # some code
        # Below line will print any print to log file as well.
        print("Creating DB Connection", file = log)
    except Exception:
        traceback.print_exc(file=log)
        continue

edited Jul 07 '20 at 05:32

slow-but-steady

961
9
15

answered Oct 24 '17 at 06:58

Avinav Mishra

718
9
12

1

A `continue` statement can't be used outside of loops afaik. Aside from that, it works perfectly. Thanks! – SomeDude Sep 07 '19 at 09:09
2

use 'pass' , no 'continue' – Maurício Morhy Jun 23 '21 at 05:10

score 4 · Answer 4 · edited Apr 10 '19 at 12:13

4

This catches everything. But it is much, much better to catch the exact exception. python <= 2.7

while True:
  try:
    doStuff()
  except Exception, e:
    f = open('log.txt', 'w')
    f.write('An exceptional thing happed - %s' % e)
    f.close()

edited Apr 10 '19 at 12:13

Joe

12,057
5
39
55

answered Aug 01 '10 at 23:58

nate c

8,802
2
27
28

score 0 · Answer 5 · answered Feb 12 '23 at 02:39

Add a continue statement in the except block. The continue statement will continue the loop at the next item, bypassing any remaining code within the current loop iteration.

for p in query:

    try:
        # The thing that might fail
    except IOError as error:
        print(error)
        continue

For more information, see: How To Use Break, Continue, and Pass Statements when Working with Loops in Python 3

How to log error to file, and not fail on exception

5 Answers5

Linked