How to use IPython.parallel for functions with multiple inputs?

Question

This is my first attempt at using IPython.parallel so please bear with me.

I read this question Parfor for Python and am having trouble implementing a simple example as follows:

import gmpy2 as gm
import numpy as np
from IPython.parallel import Client

rc = Client()
lview = rc.load_balanced_view()
lview.block = True

a = 1
def L2(ii,jj):
    out = []
    out.append(gm.fac(ii+jj+a))
    return out 

Nloop = 100
ii = range(Nloop)
jj = range(Nloop)
R2 = lview.map(L2, zip(ii, jj))

The problems I have are:

1. a is defined outside the loop and I think I need to do something like "push" but am a bit confused by that. Do I need to "pull" after?
2. there are two arguments that are required for the function and I don't know how to pass them correctly. I tried things like zip(ii,jj) but got some errors.
3. Also,, I assume the fact that I'm using a random library gmpy2 shouldn't affect things. Is this correct? Do I need to do anything special for this?

Ideally I would like your help so on this simple example the code runs error free.

If you think it would be beneficial to post my failed attempts at #2 let me know. I'm in the dark with #1.

Answer 1

I found two ways that make this work:

One is pushing the variable to the cores. There is no need to pull it. The variable will simply be defined in the namespace of each process-engine.

rc.client[:].push({'a':a})
R2 = lview.map(L2, ii, jj)

The other way is as to redefine L2 to take a as an input and pass an array of a's to the map function:

def L2(ii,jj,a):
    out = []
    out.append(gm.fac(ii+jj+a))
    return out
R2 = lview.map(L2, ii, jj, [a]*Nloop)

With regards to the import as per this website: http://ipython.org/ipython-doc/dev/parallel/parallel_multiengine.html#non-blocking-execution You simply import the required libraries in the function:

Note the import inside the function. This is a common model, to ensure that the appropriate modules are imported where the task is run. You can also manually import modules into the engine(s) namespace(s) via view.execute('import numpy')().

Or you can do as per this link http://ipython.org/ipython-doc/dev/parallel/parallel_multiengine.html#remote-imports