1

I have a function that should take a list of arguments and distribute it to a set number of threads for them to process it. But I have not yet found a quick and easy to implement way to split the arguments into smaller sub lists not of a set length but rather a predetermined number. I looked at this post, but the answers describe splitting into lists of a set length if I am not mistaken. The function I want to implement this takes a list as parameter and has access to the current number of threads

I tried a method of splitting into list of set length where I just used len(list) / nr_of_threads, but I don't know how to efficiently handle cases where there might be to many / not enough sublists for every thread

My Question (tl;dr): Is there a very fast and efficient way of splitting a list into a set number of sub lists of about equal length?

randomGuy
  • 33
  • 5
  • 1
    After splitting into smaller chunks, you should initailize a queue and after then you should pass element to func and pop that element from queue for each of your new thread. – anlgrses Aug 10 '20 at 11:40
  • I could try that, but then I probably want as small of a chunk size as possible (to equalize the amount of data per thread). Doesn't that mean quite a bit of overhead, since I have to repeatedly call the function with new data? – randomGuy Aug 10 '20 at 11:44
  • 1
    There is no other way to optimize it. You have to give each chunk to new thread. It's a thread not an another process. That means your variable isn't copied. Therefore, the complexity and memory usage is the same – anlgrses Aug 10 '20 at 11:50
  • I just used a shared list for all the threads, it works fine. Thank you anyways! – randomGuy Aug 10 '20 at 12:12

0 Answers0