3

I have a program that splits the processing of a large dataset into parallel processes with billiard like this:

    import billiard 
    from billiard import Queue


    def process_cursor(skip_n, limit_n, out_q):
        # ... lines of code ...
        # potentially long method call that should return after max. 5 seconds
        result = possible_long_running_function(arguments)
        # ... more lines of code ...
        out_q.put(function_result)


    batch_size = round(collection_size / n_cores + 0.5)
    skips = range(0, n_cores * batch_size, batch_size)

    out_q = Queue()
    processes = [billiard.Process(
        target=process_cursor, args=(skip_n, batch_size, out_q)) for skip_n in skips]

    for process in processes:
        process.start()

    results = []
    for i in range(n_cores):
        results.append(out_q.get())

    for process in processes:
        process.join()

As part of the processing in each thread, a line of code executes a function that can take too long to be executed.

result = possible_long_running_function(input)

How could I throw an Exception if the function takes longer than a given time. Could the process create a subprocess and kill it if takes too long?

dsalfran
  • 31
  • 4

0 Answers0