I have a program that splits the processing of a large dataset into parallel processes with billiard like this:
import billiard
from billiard import Queue
def process_cursor(skip_n, limit_n, out_q):
# ... lines of code ...
# potentially long method call that should return after max. 5 seconds
result = possible_long_running_function(arguments)
# ... more lines of code ...
out_q.put(function_result)
batch_size = round(collection_size / n_cores + 0.5)
skips = range(0, n_cores * batch_size, batch_size)
out_q = Queue()
processes = [billiard.Process(
target=process_cursor, args=(skip_n, batch_size, out_q)) for skip_n in skips]
for process in processes:
process.start()
results = []
for i in range(n_cores):
results.append(out_q.get())
for process in processes:
process.join()
As part of the processing in each thread, a line of code executes a function that can take too long to be executed.
result = possible_long_running_function(input)
How could I throw an Exception
if the function takes longer than a given time. Could the process create a subprocess and kill it if takes too long?