I have a function that generates a lambda and places it into a dictionary, along these lines:
class Example:
def __init__(self):
self.functions = {}
def create_function(self, key):
(code that sets up parameters)
func = scipy.interpolate.interp2d(x, y, z) # returns a function for interpolation
self.functions[key] = lambda a, b: func(b, a) # axes need to be switched
In a different part of the class, this function is called in a loop that I'd like to parallelize, as it takes ~30 seconds to run in serial. I've tried using multiprocessing's Pool, but that can't write to the dictionary. I've also tried modifying the function to return the lambda in a couple ways, but it can't return a lambda, saying AttributeError("Can't pickle local object '(class).(function).<locals>.<lambda>'").
The below code segments would be in a method of Example, and are approximately what I tried.
pool = multiprocessing.Pool()
keys = [] # list of keys it needs to operate on
(code to set up keys)
# Doesn't work, won't write to dictionary (Effectively does nothing but take up time)
pool.map(self.create_function, keys)
# Doesn't work, can't return lambda (AttributeError)
new_dict = dict(pool.map(self.create_function, keys))
self.functions.update(new_dict)
Is there a way to parallelize this?
EDIT:
I was able to solve the problem as relevant to my project by separating the (code that sets up parameters) part and paralellizing that instead, but the question is still relevant - how can a function that returns a lambda be parallelized?
