3

I am working on Google Colab environment to create a Siamese network using Keras to verify the images. I have used this code from GitHub. But I get an error when I tried to run the pickle.dump code: 

with open(os.path.join(save_path,"train.pickle"), "wb") as f:
    pickle.dump((X,c),f)

The error message is:

---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-7-af9d0618d385> in <module>()
      3 
      4 with open(os.path.join(save_path,"train.pickle"), "wb") as f:
----> 5         pickle.dump((X,c),f)
      6 
      7 

OverflowError: cannot serialize a bytes object larger than 4 GiB

I found some related questions in this website, but I could not find a useful answer. How can I solve this error?

Noran
  • 707
  • 1
  • 8
  • 19

1 Answers1

8

Use pickle with protocol=4, e.g.,

pickle.dump((X,c), f, protocol=4)
Bob Smith
  • 36,107
  • 11
  • 98
  • 91
  • I tried this code several times, but the session stopped during running this code and I got this message "Session crash for an unknown reason " – Noran Mar 10 '19 at 20:02