2

I've just begun learning concurrent programming in Python.

I created a function which takes one positional argument and then **kwargs.

Code:

def download_from_api(api_client_method, **kwargs):
    try:
        response = api_client_method(**kwargs)
        return response
    except api.exceptions.APIException as e:
        print(f'{kwargs}: {api_client_method.__name__} - {e})

I'd like to pass this function to concurrent.futures.ThreadPoolExecutor().map() object.

Below is the code i used for trying, passing everything as positional args, but it didnt work:

these_should_be_passed_as_keyword_args = ['A', 'B']
n = len(these_should_be_passed_as_keyword_args)
functions = [api_client.relevant_method] * n

with concurrent.futures.ThreadPoolExecutor() as executor:
    results = executor.map(download_from_api, functions, these_should_be_passed_as_keyword_args)

Any help is really appreciated.

Thank you!

Raul Chiarella
  • 518
  • 1
  • 8
  • 25
DavidLojkasek
  • 25
  • 5

1 Answers1

1

There are some good examples here: Pass multiple parameters to concurrent.futures.Executor.map?

Alternatively, I prefer using .submit() Here are the docs, and here is a tutorial

As I cant execute your download_from_api function, here is a basic example with a function that just prints random generated kwargs:

import string
import random
import concurrent.futures as cf


def your_function(argument, **kwargs):
    print(argument, kwargs)


random_kwargs = [{
    ''.join(random.choice(string.ascii_lowercase) for _ in range(4)):
        ''.join(random.choice(string.ascii_lowercase) for _ in range(4))
} for _ in range(10)]

with cf.ThreadPoolExecutor() as executor:
    futures = [executor.submit(your_function, argument, **kwargs) 
               for argument, kwargs in enumerate(random_kwargs)]
bitflip
  • 3,436
  • 1
  • 3
  • 22
  • Thank you! I'll have to study the difference between `map()` and `submit()` more. I started with `map()` as I read it's usually better when converting `for` loops. – DavidLojkasek Sep 25 '22 at 19:56
  • Especially when working with an API you probably want to send a lot of requests and then work with them as soon as you get the data. Another useful function in that context is `for future in cf.as_completed(futures)`that lets you loop over the submitted futures and collect their data whenever they are completed. – bitflip Sep 25 '22 at 19:59