0

Trying to use stackless python (2.7.2) with SPickle to send a test method over celery for execution on a different machine. I would like the test method (code) to be included with the pickle and not forced to exist on the executing machines python path.

Been referencing following presentation: https://ep2012.europython.eu/conference/talks/advanced-pickling-with-stackless-python-and-spickle

Trying to use the technique shown in the checkpointing slide 11. The RPC example doesn't seem right given that we are using celery:

Client code:

from stackless import run, schedule, tasklet
from sPickle import SPickleTools


def test_method():
    print "hello from test method"

tasks = []
test_tasklet = tasklet(test_method)()
tasks.append(test_tasklet)

pt = SPickleTools(serializeableModules=['__test_method__'])
pickled_task = pt.dumps(tasks)

Server code:

pt = sPickle.SPickleTools()
unpickledTasks = pt.loads(pickled_task)

Results in:

[2012-03-09 14:24:59,104: ERROR/MainProcess] Task    
celery_tasks.test_exec_method[8f462bd6-7952-4aa1-9adc-d84ee4a51ea6] raised exception:   
AttributeError("'module'
object has no attribute 'test_method'",)
Traceback (most recent call last):
File "c:\Python27\lib\site-packages\celery\execute\trace.py", line 153, in trace_task
R = retval = task(*args, **kwargs)
File "c:\Python27\celery_tasks.py", line 16, in test_exec_method
unpickledTasks = pt.loads(pickled_task)
File "c:\Python27\lib\site-packages\sPickle\_sPickle.py", line 946, in loads
return unpickler.load()
AttributeError: 'module' object has no attribute 'test_method'

Any suggestions on what I am doing incorrect or if this is even possible?

Alternative suggestions for doing dynamic module loading in a celeryd would also be good (as an alternative for using sPickle). I have experimented with doing:

py_mod = imp.load_source(module_name,'some script path')
sys.modules.setdefault(module_name,py_mod)

but the dynamically loaded module does not seem to persist through different calls to celeryd, i.e. different remote calls.

Ryan R.
  • 2,478
  • 5
  • 27
  • 48

1 Answers1

1

You must define test_method within its own module. Currently sPickle detects whether test_method is defined in a module that can be imported. An alternative way is to set the __module__ attribute of the function to None.

def test_method():
    pass

test_method.__module__ = None
Dhara
  • 6,587
  • 2
  • 31
  • 46
Anselm Kruis
  • 11
  • 1
  • Would you agree that there is currently no built in way with sPickle to bundle module dependencies and dynamically load on a remote machine? I believe when I was evaluating this that I was hoping sPickle would do dynamic module dependency bundling and sandboxing, e.g. http://stackoverflow.com/questions/10099326/how-to-do-an-embedded-python-module-for-remote-sandbox-execution. Will give you the check if you agree. – Ryan R. Jun 25 '12 at 19:37