Continue running code if one aspect of a try block fails, but still raise the exception

Question

As part of a much larger script to process excel files into different formats with multiple Python functions, I have this function to actually run all the other functions:

print = logging.info
class MyException(Exception):
    pass
@catch_exception_and_send_email
def validation_and_processing(file_list):
    '''
    This function the file list and runs each file in the directory through the validation and processing functions.
    
    args:
        file_list : output file path     
    '''
    exceptions = []
    for file in files:
        try:
            if file.startswith('Nomura'):
                check_filename_format(input_file = file)
                nomura = nomura_validation(input_dir = input_dir, file = file)
                nomura_df = nomura_processing(nomura)
                write_output(path=output_dir, filename=file, dataframe=nomura_df)
                shutil.move(input_dir+file, archive_dir+file)
                print("Congratulations! {} dealer file processed successfully on {} at {}".format(file, date, time))

            elif file.startswith('MSTN'):
                check_filename_format(input_file = file)
                morgan = morgan_validation(input_dir = input_dir, file = file) 
                morgan_df = morgan_processing(morgan)
                write_output(path=output_dir, filename=file, dataframe=morgan_df)
                shutil.move(input_dir+file, archive_dir+file)
                print("Congratulations! {} dealer file processed successfully on {} at {}".format(file, date, time))

            elif file.startswith('JPM'):
                check_filename_format(input_file = file)
                jpm, input_file = jpm_validation(input_dir = input_dir, file = file)
                jpm = jpm_processing(jpm, input_file)
                write_output(path=output_dir, filename=file, dataframe=jpm)
                shutil.move(input_dir+file, archive_dir+file)
                print("Congratulations! {} dealer file processed successfully on {} at {}".format(file, date, time))

            elif file.startswith('BofA'):
                check_filename_format(input_file = file)
                bofa_not_ginnie, bofa_ginnie = bofa_validation(input_dir=input_dir, file=file)
                bofa_final = bofa_processing(df1=bofa_not_ginnie, df2=bofa_ginnie, input_file=file)       
                write_output(path=output_dir, filename=file, dataframe=bofa_final)
                shutil.move(input_dir+file, archive_dir+file)
                print("Congratulations! {} dealer file processed successfully on {} at {}".format(file, date, time))

            elif file.startswith('CITI'):
                check_filename_format(input_file = file)
                citi = citi_validation(input_dir=input_dir, file=file)
                citi_final = citi_processing(df=citi, input_file=file)    
                write_output(path=output_dir, filename=file, dataframe=citi_final)
                shutil.move(input_dir+file, archive_dir+file)
                print("Congratulations! {} dealer file processed successfully on {} at {}".format(file, date, time))
                
        except (OSError, IOError, MyException):
            # This should cast a wide net since:
            # In 3.3, IOError became an alias for OSError, 
            # and FileNotFoundError is a subclass of OSError
            # also catches PermissionError
            # Source: https://stackoverflow.com/questions/15032108/pythons-open-throws-different-errors-for-file-not-found-how-to-handle-b/15032444#15032444
            # Logs the error appropriately.
            logging.error("Error with file:" + file)
            continue
        except Exception as e:
            # Reraise exception adding info about which file
            pass
            raise Exception("Error with file:", file) from e
    return exceptions

The idea is if one of the file's function fails, it should raise an exception, but should still make sure to try to run all the other files. When I run a test case with one of the files being incorrectly formatted, it raises the error for that file and successfully completes the files that occur in the directory before that file, but not the ones after. How can I modify my code to accomplish both the error being raised if it occurs (I need the code to ultimately fail if one file is incorrect because an email is sent for exceptions and a log file is updated with the errors), but still run the other files if they are correct.

Here's my code for building the log file. I use catch_exception_and_send_email as a decorator function for other functions in the script in case they raise an exception.

now = datetime.now()
date = now.strftime("%m_%d_%Y") 
time = now.strftime("%H_%M_%S")
log_filename = "log_" + date + "_" + time + ".txt"

#Initialize log file
logging.basicConfig(filename=log_filename, level=logging.DEBUG, format='%(asctime)s %(levelname)s %(name)s %(message)s', force = True)
logger=logging.getLogger(__name__)
logging.getLogger('PIL').setLevel(logging.WARNING)

'''
The below function is a decorator function that checks every function in the file at runtime for exceptions,
and updates the log file and sends an email with details of exception
'''

def catch_exception_and_send_email(func):
    def exception_catcher(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except Exception as e:
            print(e)
            send_error_email(exception=e, message="An exception {} occurred in {}".format(e, func.__name__))
            logging.exception('Oh no! {} On {} at {} in: {} function.'.format(e, date, time, func.__name__), exc_info = False)
            sys.exit(1)
    return exception_catcher

Oh, that's a good question. Ideally there would be an exception/error thrown for each failed file. — Hefe, Sep 07 '22 at 14:40
So you could return a list of exceptions with their filenames? Or an empty list if no failures? — quamrana, Sep 07 '22 at 14:41
Sure, that's a good idea. How do I append exceptions to a list? — Hefe, Sep 07 '22 at 14:42
[Bare Exceptions](https://stackoverflow.com/questions/4990718/how-can-i-write-a-try-except-block-that-catches-all-exceptions) are bad. What you could do is enumerate several exceptions which are reported and continue after reporting when they occur as in [Error/ Exception handling in for loop](https://stackoverflow.com/questions/51199436/error-exception-handling-in-for-loop-python/51200829#51200829). An error is raised with other exceptions which halt the for loop. — DarrylG, Sep 07 '22 at 14:47
@DarrylG sounds good, could you post an answer describing how to do that? — Hefe, Sep 07 '22 at 14:52

score 1 · Answer 1 · answered Sep 07 '22 at 14:47

1

You need to be able to just retain the exception:

def validation_and_processing(file_list):
    '''
    This function the file list and runs each file in the directory through the validation and processing functions.
    
    args:
        file_list : output file path     
    '''
    exceptions = []
    for file in files:
        try:
            if file.startswith('Nomura'):
                ...
                ...

        except Exception as e:      
            exceptions.append(e)

    return exceptions

answered Sep 07 '22 at 14:47

quamrana

37,849
12
53
71

I've tried this out with a test-case of one bad file, and it still only runs all the files in the directory before that one. – Hefe Sep 07 '22 at 15:05
Ok, so I've noticed that the function you supply has the parameter `file_list` (which I presume is what this function should iterate over). Do you have another loop which goes over multiple directories and calls this function? – quamrana Sep 07 '22 at 15:08

DarrylG · Answer 2 · 2022-09-07T15:41:19.640

1

Goal is to:

log expected exceptions for files and continue processing
raise exceptions and stop for other exceptions

Code

import traceback
import logging

def validation_and_processing(file_list):
    '''
    This function the file list and runs each file in the directory through the validation and processing functions.
    
    args:
        file_list : output file path     
    '''
    for file in files:
        try:
            if file.startswith('Nomura'):
                ...
                ...
        except (OSError, IOError, CustomExceptionName):
            # This should cast a wide net since:
            # In 3.3, IOError became an alias for OSError, 
            # and FileNotFoundError is a subclass of OSError
            # also catches PermissionError
            # Source: https://stackoverflow.com/questions/15032108/pythons-open-throws-different-errors-for-file-not-found-how-to-handle-b/15032444#15032444
            # CustomExceptionName class similar to: https://stackoverflow.com/questions/1319615/proper-way-to-declare-custom-exceptions-in-modern-python/53469898#53469898 
            # Logs the error appropriately.
            logging.exception("Error with file:" + file)
            continue     # continue processing

       except Exception as e:
            # Reraise exception adding info about which file
            raise Exception("Error with file:", file) from e

edited Sep 07 '22 at 15:41

answered Sep 07 '22 at 15:10

DarrylG

16,732
2
17
23

Awesome, will try this out. I also added the code to show my log file instantiation and use of that function as a decorator. I'm not sure if that changes the need for the logging function you call. – Hefe Sep 07 '22 at 15:13
@Hefe -- your tweaks sound even better. – DarrylG Sep 07 '22 at 15:14
For sure, so are you saying I don't need your `except (OSError, IOError) as e` block? Also, with or without that the code still stops at the broken file – Hefe Sep 07 '22 at 15:18
I was saying you don't need the `as e` part. If the code is still stopping, do you know what exception it is reporting? Is it something that is not an OSError or an IOError? – DarrylG Sep 07 '22 at 15:21
Okay, gotcha. The code does stop but it pulls an exception that I wrote in another function not included here. Here's an example of the exception: `Exception: The PUBLISH sheet of file: C:/Users/JWeinstein/Downloads/Sample Files/Input/JPM2022_05_06.xlsx's column 'LT' are not in expected format.` – Hefe Sep 07 '22 at 15:22
@Hefe if make your custom exception a class then it could be added to the list of exceptions to just log and continue. See [Proper way to declare custom exceptions in modern Python](https://stackoverflow.com/questions/1319615/proper-way-to-declare-custom-exceptions-in-modern-python/53469898#53469898) – DarrylG Sep 07 '22 at 15:28
Hmm, interesting. Thanks for sharing! – Hefe Sep 07 '22 at 15:36
@Hefe modified code to illustrate. – DarrylG Sep 07 '22 at 15:41
Modified my code to include those changes, still running into the same issue with the code stopping at the broken file – Hefe Sep 07 '22 at 15:52
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/247851/discussion-between-darrylg-and-hefe). – DarrylG Sep 07 '22 at 16:01

Continue running code if one aspect of a try block fails, but still raise the exception

2 Answers2