6

I am attempting to run a job using the ipyparallel system with lru_cache and am running into issues.

From the terminal:

ipcluster start -n 2

In an ipython notebook:

from ipyparallel import Client
clients = Client()

def world():
    return hello() + " World"

def hello():
    return "Hello"

world()
'Hello World'

Running it with ipyparallel requires this:

clients[:].push(dict(hello=hello))

Without the previous line the following fails, which is not unexpected, but works fine if it is run:

clients[:].apply_sync(world)
['Hello World', 'Hello World']

This all works as expected, however, with the lru_cache the parallel step generates errors

from ipyparallel import Client
from functools import lru_cache

clients = Client()

def world():
    return hello() + " World"

@lru_cache(maxsize=2048)
def hello():
    return "Hello"

clients[:].push(dict(hello=hello))
clients[:].apply_sync(world)

This fails with the following error:

  [0:apply]: 
---------------------------------------------------------------------------NameError                                 
Traceback (most recent call last)<string> in <module>()
<ipython-input-17-9ac5ef032485> in world()
NameError: name 'hello' is not defined

[1:apply]: 
---------------------------------------------------------------------------NameError                                 
Traceback (most recent call last)<string> in <module>()
<ipython-input-17-9ac5ef032485> in world()
NameError: name 'hello' is not defined

I realize this may be a namespace issue where lru_cache is using a closure, however I'm hoping there is a workaround so that I am able to use it. Or if someone can tell me that this simply isn't possible that would be useful as well, thanks.

I am using:
ipykernel (4.1.1)
ipyparallel (4.1.0)
ipython (4.0.1)
Python (3.5.1)

johnchase
  • 13,155
  • 6
  • 38
  • 64
  • I'm pretty sure it has to do with the decorator itself. If you replace `lru_cache` with any other decorator function you will have the same error. I don't know if it can be solved though.. – Imanol Luengo Jan 06 '16 at 19:14
  • Thanks, @imaluengo that was my initial thought as well, though I was able to get it working with other decorators. I debated about including an example with a working decorator in the original post though it was getting to be quite a long post. – johnchase Jan 06 '16 at 19:55
  • Ops, I tried with a test decorator in python 2.7 and it didn't work for me :/ – Imanol Luengo Jan 06 '16 at 20:05

0 Answers0