1

As a follow up to this question: How to pickle a python function with its dependencies?

What is a good approach for determining a method's dependencies? For instance, similar to the above post, if I have a function f that uses methods g and y is there an easy way to get a reference to g and y dynamically?

Further, I guess you would want this method to recurse down the entire function graph such that if y depended on z you could also bundle up z.

I see that disco uses the following module for this: https://github.com/discoproject/disco/blob/master/lib/disco/worker/classic/modutil.py

Any other suggestions on how to tackle this? The disco approach seems to be module based so you potentially have to bundle up more than you need to actually execute the root method.

Community
  • 1
  • 1
Ryan R.
  • 2,478
  • 5
  • 27
  • 48

1 Answers1

1

To do this, I'd use dill, which can serialize almost anything in python. Dill also has some good tools for helping you understand what is causing your pickling to fail when your code fails.

>>> import dill
>>> dill.loads(dill.dumps(your_bad_object))
>>> ...
>>> # if you get a pickling error, use dill's tools to figure out a workaround
>>> dill.detect.badobjects(your_bad_object, depth=0)
>>> dill.detect.badobjects(your_bad_object, depth=1)
>>> ...

If you absolutely wanted to, you could use dill's badobjects (or one of the other detection functions) to dive recursively into your object's reference chain, and pop out the unpickleable objects, instead of calling it at at every depth, as above.

Also, objgraph is a pretty handy compliment to the test suite too.

>>> # visualize the references in your bad objects
>>> objgraph.show_refs(your_bad_object, filename='your_bad_object.png')

Or, as I mentioned in your above noted post, you can use dill to pickle the entire python session in one command. That's a bit overkill for your question here, but it'd work too.

Mike McKerns
  • 33,715
  • 8
  • 119
  • 139