4

Here's a snippet that will succeed in serializing with dill, but fail with pickle. It is surprising that Tensor objects aren't natively pickleable. Is this a fundamental limitation of thread-aware Tensors, or is it just not implemented?

import dill
import pickle
import tensorflow as tf

dill.dumps(tf.zeros((1,1)))
print("Dill succeeded")
pickle.dumps(tf.zeros((1,1)))
print("Pickle succeeded")

Output:

$ python foo.py
Dill succeeded
Traceback (most recent call last):
  File "foo.py", line 7, in <module>
    pickle.dumps(tf.zeros((1,1)))
TypeError: can't pickle _thread.lock objects
denfromufa
  • 5,610
  • 13
  • 81
  • 138
eqzx
  • 5,323
  • 4
  • 37
  • 54
  • Possible duplicate http://stackoverflow.com/questions/39286665/how-to-avoid-pickling-errors-when-sharing-objects-between-threads – DYZ May 03 '17 at 20:28

2 Answers2

3

The reason why dill can serialize these objects, but not pickle? Simple answer is that pickle cannot serialize most objects in python, the thread.lock object included. If you want to serialize one of these objects, use an advanced serialization library like dill. As to exactly why pickle can't, I think originally it stems from the implementation of the GIL and the frame object rendering some objects unserializable, and thus there was no drive to serialize everything in the language. There's always been talk about security issues stemming from serialization of all python objects, but I think that's a red herring. Not having full language serialization limits the ability to operate in parallel computing, so hopefully pickle will learn from dill how to serialize more objects.

denfromufa
  • 5,610
  • 13
  • 81
  • 138
Mike McKerns
  • 33,715
  • 8
  • 119
  • 139
2

I find that I am unable to reproduce the output of eqzx's script and serializing fails for both dill and pickle.

While this question is several years old, I believe the issue wasn't with pickle but rather the fact that tensors in tensorflow 1.x are not evaluated immediately. They are instead evaluated when their graph is executed. (In tensorflow 2.0 eager execution is enabled by default so you don't really have to deal with this paradigm).

Using tf.enable_eager_execution() or evaluating a tensor within a tf.Session() context

The following script doesn't throw an error when using either Pickle or Dill:

import tensorflow as tf
tf.enable_eager_execution()
import pickle 
# import dill

n = tf.zeros((1,1))
pickle.dumps(n)
# dill.dumps(n)

print('Success')

Same script using Session.run() instead doesn't throw an error:

import tensorflow as tf
import pickle
# import dill

with tf.Session() as sess:
    n = sess.run(tf.zeros((1,1)))
pickle.dumps(n)
# dill.dumps(n)

print('Success')
marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
evantkchong
  • 2,251
  • 3
  • 14
  • 29
  • 1
    If you look at the traceback, it's obvious that the `Lock` object is not pickling. Note that when you say the functions are not evaluated immediately, that means that they are put on the execution stack, which means there's a connection to the `frame` object as noted in my answer. Both `dill` and `Tensor` objects have evolved significantly since the OP posted. – Mike McKerns Oct 30 '19 at 02:43
  • Hi @MikeMcKerns, while I'm not entirely familar with the inner workings of tensorflow nor how the `frame` object is used by it, I was answering based on the docs for a tf.Tensor. A tf.Tensor in the build context doesn't actually contain the values for the tensor but rather is handle for the outputs for a tensorflow operation. As it isn't a Python object, I didn't think it made sense for it to be serializable unless it was evaluated. – evantkchong Nov 05 '19 at 03:19
  • 1
    That may be true... The objects that makeup a python function are different before and after the first evaluation of the function. With lazy evaluation, a python function's skeleton is composed when it's built, but the contents of it aren't handled until it's called the first time. Before that, the guts of a function are put on the execution stack (essentially on the `frame`) to be retrieved when it's called. Also, note that essentially everything in python is an object -- a list, a file handle... basically anything that has a `type` (i.e. derives from a python class). – Mike McKerns Nov 05 '19 at 13:20
  • @MikeMcKerns I hope you don't mind me asking additional questions because you seem really well versed with this topic. In that case wouldn't it matter that tensorflow ops make calls to the tensorflow C++ api when called? In fact `frame` seems to be a C++ struct. – evantkchong Nov 06 '19 at 09:29
  • Python objects are generally thin classes wrapped around C-type objects -- unless you are calling directly to one of the `ctypes` (if I'm not mistaken). So, I'd have to see how the calls to the `tensorflow` C-type api are made. – Mike McKerns Nov 06 '19 at 13:41