I'm trying to load two large dictionaries that I pickled earlier using cPickle. Here's the code that I used to create the pickle:
f_out = open("file_1.pickle", 'wb')
cPickle.dump(obj_1, f_out, protocol=-1)
f_out.close()
f_out = open("file_2.pickle", 'wb')
cPickle.dump(obj_2, f_out, protocol=-1)
f_out.close()
f_out is not used anywhere else in the script that creates the dictionaries, and the script ran without errors. The pickles are large: 895.9 MB for obj_1 and 978.3 MB for obj_2. The files have been renamed, but not altered in any other way. The following code is used to load the pickles:
f1 = open("file_1_renamed.pickle", 'rb')
obj_1 = cPickle.load(f1)
f1.close()
f2 = open("file_2_renamed.pickle", 'rb')
obj_2 = cPickle.load(f2)
f2.close()
obj_1 loads without issues, but trying to load obj_2 creates an EOFError. I get the same error with pickle.load() and also when I open the file as text, not as binary. Another question on Stack Overflow (Python Pickling Dictionary EOFError) suggests that this might be related to insufficient memory, and but I'm not entirely convinced – why would a memory issue lead to an end-of-file error? I'm using OS X 10.9.5 and Python 2.7.8.
