27

From https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor.map

If a func call raises an exception, then that exception will be raised when its value is retrieved from the iterator.

The following snippet only outputs the first exception (Exception: 1), and stops. Does this contradict the above statement? I expect the following to print out all exceptions in the loop.

def test_func(val):
    raise Exception(val)

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    for r in executor.map(test_func,[1,2,3,4,5]):
        try:
            print r
        except Exception as exc:
            print 'generated an exception: %s' % (exc)
Josh Correia
  • 3,807
  • 3
  • 33
  • 50
user1008636
  • 2,989
  • 11
  • 31
  • 45

4 Answers4

27

Ehsan's solution is good, but it may be slightly more efficient to take the results as the are completed instead of waiting for sequential items in the list to finish. Here is an example from the library docs.

import concurrent.futures
import urllib.request

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

# Retrieve a single page and report the URL and contents
def load_url(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()

# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    # Start the load operations and mark each future with its URL
    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
    for future in concurrent.futures.as_completed(future_to_url):
        url = future_to_url[future]
        try:
            data = future.result()
        except Exception as exc:
            print('%r generated an exception: %s' % (url, exc))
        else:
            print('%r page is %d bytes' % (url, len(data)))
leafmeal
  • 1,824
  • 15
  • 15
16

As mentioned above, unfortunately executor.map's API is limited and only lets you get the first exception. Also, when iterating through the results, you will only get values up to the first exception.

To answer your question, if you don't wan to use a different library, you can unroll your map and manually apply each function:

future_list = []
with concurrent.futures.ThreadPoolExecutor() as executor:
  for arg in range(10):
    future = executor.submit(test_func, arg)
    future_list.append(future)

for future in future_list:
  try:
    print(future.result())
  except Exception as e:
    print(e)

This allows you to handle each future individually.

Ehsan Kia
  • 1,475
  • 18
  • 26
7

The map method returns a generator which allows to iterate through the results once ready.

Unfortunately, it is not possible to resume a generator after an exception occurs. From PEP 255.

If an unhandled exception-- including, but not limited to, StopIteration --is raised by, or passes through, a generator function, then the exception is passed on to the caller in the usual way, and subsequent attempts to resume the generator function raise StopIteration. In other words, an unhandled exception terminates a generator's useful life.

There are other libraries such as pebble which allow to continue the iteration after an error occurs. Check the examples in the documentation.

noxdafox
  • 14,439
  • 4
  • 33
  • 45
  • thanks. Is pebble is common framework used in python for multithreading and multiprocessing? Isn't it redundant now that python has its own native concurrent.futures module ? – user1008636 Jun 28 '18 at 13:06
  • `pebble` overcomes some limitation of Python's native library such as the above example and other issues like terminating timing out tasks. – noxdafox Jun 28 '18 at 13:14
  • Doesn't concurrent.futures have options to do such things now ? – user1008636 Jun 28 '18 at 13:19
  • No it doesn't. That's why `pebble` was designed. Another alternative library worth to check is [`billiard`](https://github.com/celery/billiard). – noxdafox Jun 28 '18 at 20:24
  • What an odd behavior... I wonder why is it like this... – Anton Daneyko Sep 30 '20 at 16:18
5

Although others gave excellent answers about the proper way to catch multiple exceptions, I would like to answer why the way, in the question, to catch the exception is wrong. The following snippet:

class ExceptionA(Exception):
    pass


def test_func(val):
    raise ExceptionA(val)


with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    try:
        for r in executor.map(test_func, [1, 2, 3, 4, 5]):
            try:
                print(r)
            except ExceptionA as exc:
                print(f'Catch inside: {exc}')

    except ExceptionA as exc:
        print(f'Catch outside: {exc}')

gives output Catch outside: 1.

The python docs reads:

If a func call raises an exception, then that exception will be raised when its value is retrieved from the iterator.

This means if you want to catch the exception, you need to catch it outside of the loop, since the value is retrieved at the loop statement instead of the print statement.

Tengerye
  • 1,796
  • 1
  • 23
  • 46