0

I realize a form of this question has already been asked several times on Stackoverflow but it really seems like this situation is different.

I cannot open a pickled file due to an EOFError but the file is certainly not empty since it 25 megs large. Further, I find it almost impossible that there is something fundamentally wrong with the file.

I have been using pickles extensively over the last 7 years and it's been roughly 3 years since I last encountered this error. Second, when I last saved the file I had two other pickles of basically the same form and I saved them in the same way and they are working fine.

I saved the file in the following manner:

import pickle

temp = open(name, "wb")
pickle.dump(obj, temp)
temp.close()

Where name is the file name and obj is the thing I saved. When I open it with this code:

pkl_file = open(name, 'rb')
obj = pickle.load(pkl_file)
pkl_file.close()

I get the EOFError. The thread here seems to have the most information. But part of their code they're using has the line:

num_cars = pickle.load(car_file)

Which will throw the EOFError in my case. I need to know if there is some different way to open the pickled file and get access to the info stored in it. The info is certainly there as it is composed of 25 megs. I'm also using Python 3.8.0. Thanks.

Just Mohit
  • 141
  • 1
  • 13
bobsmith76
  • 160
  • 1
  • 9
  • 26
  • https://stackoverflow.com/questions/35067957/how-to-read-pickle-file – aminrd Jun 09 '21 at 03:57
  • What's the general structure of your object? – drum Jun 09 '21 at 04:05
  • It's a dictionary with keys as strings and values as a list of strings. Like I said at the same time I saved it, I saved two other pickles of the same format and they are opening fine. – bobsmith76 Jun 09 '21 at 04:06
  • I already said that the file is 25 megs. Plus, as I said in the comment above, the only object types in my pickle are strings and lists – bobsmith76 Jun 09 '21 at 04:37
  • As I said in the above comment: 'It's a dictionary with keys as strings and values as a list of strings'. – bobsmith76 Jun 09 '21 at 05:34
  • If that is the case maybe there is some upper limit to the size of the objects being pickled. I don’t know of any but it might be worth seeing if you could somehow break up the object. Similar to that question you quoted maybe it’s possible to pickle key, value pairs and pickle them, terminating with a None, then reverse the process to reconstruct them. Can’t think of any other reason why the load is failing. – Ben Y Jun 09 '21 at 08:33
  • It's actually one of the smaller pickles I've worked with. It only has 35,000 keys and each value has maybe 200 strings. It's only 25 megs and the other pickled dicts are opening just fine. Plus, I wouldn't know how to break up the object, seeing as I can't open it – bobsmith76 Jun 09 '21 at 08:44
  • I saw that memory could affect the unpickling, as discussed here: https://stackoverflow.com/questions/10263564/python-pickling-dictionary-eoferror I know you said it worked for other files, it's hard to tell when memory might be low. The only other possible solution I could offer you is to try a newer version of 3.8, but beside that, not a whole lot else I can recommend. My apologies for making you repeat your question several times, as I was trying to find a solution for you. – Ben Y Jun 09 '21 at 18:19
  • no problem. thanks for your help. i've already given up on the problem. I have no idea what I did wrong and I have no way of repeating the error. – bobsmith76 Jun 10 '21 at 00:15

0 Answers0