0

I am trying to read a file using the following code.

precomputed = pickle.load(open('test/vgg16_features.p', 'rb'))
features = precomputed['features']

But getting this error.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 5: ordinal not in range(128)

The file I am trying to read contains image features which are extracted using deep neural networks. The file content looks like below.

(dp0
S'imageIds'
p1
(lp2
I262145
aI131074
aI131075
aI393221
aI393223
aI393224
aI524297
aI393227
aI393228
aI262146
aI393230
aI262159
aI524291
aI322975
aI131093
aI524311
....
....
....

Please note that, this is big file, of size 2.8GBs.

I know this is a duplicate question but I followed the suggested solutions in other stackoverflow posts but couldn't solve it. Any help would be appreciated!

Wasi Ahmad
  • 35,739
  • 32
  • 114
  • 161

1 Answers1

2

Finally I found the solution. The problem was actually about unpickling a python 2 object with python 3 which I couldn't understand first because the pickle file I got was written through a python 2 program.

Thanks to this answer which solved the problem. So, all I need to do is set the encoding parameter of pickle.load() function to latin1 because latin1 works for any input as it maps the byte values 0-255 to the first 256 Unicode codepoints directly.

So, the following worked for me!

precomputed = pickle.load(open('test/vgg16_features.p', 'rb'), encoding='latin1')
Wasi Ahmad
  • 35,739
  • 32
  • 114
  • 161