Python3 pickle serialization with Cmd

Question

I am new to Python and as my first project I am attempting to convert a Python2 script to Python3.

The script is failing when it attempts to serialize a class using pickle.

It seems as though it is failing as I am trying to save a class which uses the Cmd CLI.

This code works using Python2.

Can anyone tell me what is wrong with the script and how I fix it?

import sys
import cmd

try:
    import pickle as pickle
except:
    import pickle
import os.path

def main():    

        app = Labyrinth()
        turnfile = "turn0.lwot"
        app.Save(turnfile)

class CLI(cmd.Cmd):
    def __init__(self):
        cmd.Cmd.__init__(self)

class Labyrinth(cmd.Cmd):

    def __init__(self):
        cmd.Cmd.__init__(self)

    def Save(self, fname):
        with open(fname, 'wb') as f: 
            pickle.dump(self,f, 2)
        f.close()
        print ("Save Successful!")
        sys.exit()

if __name__ == '__main__':
    main()

What is with the import pickle as pickle wrapped in a try/except ? — , Dec 26 '16 at 19:16
@Turry - I think that's a mistyped `import cPickle as pickle` for python 2. — tdelaney, Dec 26 '16 at 19:19

score 3 · Answer 1 · answered Dec 26 '16 at 19:30

Not all objects are picklable. In particular, file objects are problematic because you can't generally restore their state later. cmd.Cmd holds stdin and stdout file objects and that should make them unpicklable. I was quite surprised that it worked in python 2, but it didn't really... Even though the stdin and stdout pickled, the unpickled object you get back later doesn't work, as in this example:

>>> import sys
>>> import pickle
>>> sys.stdout.write('foo\n')
foo
>>> serialized = pickle.dumps(sys.stdout, 2)
>>> stdout = pickle.loads(serialized)
>>> stdout.write('bar\n')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: I/O operation on closed file
>>>

So, even though this bit of code didn't fail, the object shouldn't be usable later. You can add a few special methods to an object that let you fix objects so they can be serialized. Here, I've stripped the bad attributes on save and added them back on restore. Now you can pickle, unpickle and it actually works when you are done.

import sys
import cmd

try:
    import cPickle as pickle
except:
    import pickle
import os.path

def main():    

        app = Labyrinth()
        turnfile = "turn0.lwot"
        app.Save(turnfile)

class CLI(cmd.Cmd):
    def __init__(self):
        cmd.Cmd.__init__(self)

class Labyrinth(cmd.Cmd):

    def __init__(self):
        cmd.Cmd.__init__(self)

    def Save(self, fname):
        with open(fname, 'wb') as f: 
            pickle.dump(self,f, pickle.HIGHEST_PROTOCOL)
        f.close()
        print ("Save Successful!")
        sys.exit()

    def __getstate__(self):
        # stdin/out are unpicklable. We'll get new ones on load
        return tuple(((k,v) for k,v in self.__dict__.items()
            if k not in ('stdin', 'stdout')))

    def __setstate__(self, state):
        self.__dict__.update(state)
        self.stdin = sys.stdin
        self.stdout = sys.stdout


if __name__ == '__main__':
    main()

Thank you for that, it works nicely and I have learnt a fair bit about python pickling and unpickling in the process. — gasha1, Dec 26 '16 at 22:00

score 0 · Answer 2 · edited May 23 '17 at 12:33

Playing with the protocol doesn't help. The full error message (which you should have included) is:

1027:~/mypy$ python3 stack41334887.py 
Traceback (most recent call last):
  File "stack41334887.py", line 33, in <module>
    main()
  File "stack41334887.py", line 14, in main
    app.Save(turnfile)
  File "stack41334887.py", line 27, in Save
    pickle.dump(self,f, 3, fix_imports=True)
TypeError: cannot serialize '_io.TextIOWrapper' object

Python3 made some major changes in the io system. This TextIOWrapper is, I think new to Py3.

https://docs.python.org/3.1/library/io.html#io.TextIOWrapper

Can I use multiprocessing.Pool in a method of a class? also had problems serializing a TextIOWrapper.

=========

So inspireed by @tdelaney, I checked the stdin for my PY3 session:

In [1212]: sys.stdin
Out[1212]: <_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>

So that's the thing that can't be serialized.

Python3 pickle serialization with Cmd

2 Answers2