0

I try to read a checkpoint file with pyTorch

checkpoint = torch.load(xxx.ckpt)

The file was generated by a program written using python 2.7. I try to read the file using python 3.6 but get the following error

UnicodeDecodeError: 'ascii' codec can't decode byte 0x8c in position 16: ordinal not in range(128)

Is it possible to read the file without downgrade python?

Milo Lu
  • 3,176
  • 3
  • 35
  • 46

2 Answers2

2

There are some compatibility issues in pickle between Python 2.x and Python 3.x, because of the move to unicode, you're probably saving a string as part of your model and that's why you see that error.

You can follow the recommended way to save a model in Pytorch and do:

torch.save(filename, model.state_dict())

instead of saving model. Then in Python3:

model = Model() # construct a new model
model.load_state_dict(torch.load(filename))

Another way is to unpickle in Python 2 and save it to another format that is easier to transfer between Python 2 and Python 3. For example you can save the tensors of the architecture using the Pytorch-Numpy bridge and use np.savez.

You can also try to use pickle instead of torch.load and tell it to decode ASCII strings to Python3 strings

iacolippo
  • 4,133
  • 25
  • 37
1

Eventually I solve the issue by

1) create a python2 environment using anaconda

2) read the checkpoint file using pytorch, and then save it using pickle

checkpoint = torch.load("xxx.ckpt")
with open("xxx.pkl", "wb") as outfile:
    pickle.dump(checkpointfile, outfile)

3) back to the python3 environment, read the file using pickle, save the file using pytorch

pkl_file = open("xxx.pkl", "rb")
data = pickle.load(pkl_file, encoding="latin1")
torch.save(data, "xxx.ckpt")
Milo Lu
  • 3,176
  • 3
  • 35
  • 46