For loop parallel in python

Question

I am working in parallelization in python.. and I have big calculation need to be parallel. at first I have big for loop for (1000 particles for example) so my process was not independent ,and I need independent process to make it parallel. so I divided the for loop in 2 FOR loops has calculate 500 ,500 particles..and I need to run this two independent loop parallel on two different cores(processors)...so is that possible? if yes then how? please share some guidance..

for i in particle1
   some processes
   ......

 print ( something)

2nd loop

for i in particles2
    someprocess....

print (something1)

and now i want this two different process in combine

so ...

 print (something + something1)

this i am exatclty want to do.. please share idea..

Answer 1

There's two possibilities -- multithreading and multiprocessing. Multithreading is more convenient, because it allows you to automatically share the global state among all your worker threads, rather than explicitly transferring it around. But in Python, something called the "global interpreter lock" (GIL) makes it difficult for multiple threads to actually work in parallel; they tend to get blocked behind each other, with only one thread actually doing work at a time. Multiprocessing takes more setup, and you have to explicitly transfer data around (which can be expensive), but it's more effective at actually using multiple processors.

Both multithreading and multiprocessing can be leveraged by the concurrent.futures module, which gives you task-based parallelism. That might be tricky to get your head around, but once you do, it's the most efficient way to write multiprocessor-capable code.

Finally, if your for-loop is doing the same math operations on a bunch of data, you should look into numpy, and vectorizing your data. That's the most difficult approach, but it will also give you the best performance.