6

While dumping large matrix (170000*20000) as follows

 cPickle.dump(train_set,gzip.open('train.pickle.gz','wb'), cPickle.HIGHEST_PROTOCOL)

I get following error:

SystemError: error return without exception set

How would I deal in this case?

thetna
  • 6,903
  • 26
  • 79
  • 113

1 Answers1

5

cPickle can't be used for storing very large objects ( see http://bugs.python.org/issue11564 ).

You have several options:

  • split the data into chunks and store it in multiple files
  • numpy.save
  • h5py <- my favorite because of its convenient numpy-like interface, plus the data are stored in hdf5 which enables interopability with other software/languages.
  • pytables
schreon
  • 1,097
  • 11
  • 25