3

I've got this error message and I don't get its meaning.

_pickle.UnpicklingError: pickle data was truncated

My program is using multiprocessing and is running on a workstation with 8 cores and 128 Gb of RAM.

I pickle a list of objects (objects I define in another file), and this list can be quite huge (Several 100 000 of thousands of tuples at best), but also empty.

The way I pickle the data is the following:

import _pickle as pickle

with open(join(folder, file), "wb") as output:
    pickle.dump(L, output, -1) # L is the list

And I read them like this:

with open(join(folder, file), "rb") as input:
    L = pickle.load(input)

I've been using this for a while, and that's the first time the program is throwing this error.

Mathieu
  • 5,410
  • 6
  • 28
  • 55
  • Why arg -1 while dumping? – Stack Jun 18 '18 at 13:19
  • If I recall correctly that's for `HIGHEST_PROTOCOL` – Mathieu Jun 18 '18 at 13:20
  • Was the data exceptionally large this time compared to past runs? – Edward Minnix Jun 18 '18 at 13:27
  • @EdwardMinnix Definitely not, I'd say a few thousand items top. Larger I handled was a several million item list (once), took a few hours, and 80 Gb of RAM. – Mathieu Jun 18 '18 at 13:29
  • What does this error mean, and what can cause it? – Mathieu Jun 18 '18 at 13:30
  • That error is caused by the `void bad_readline();` function in the C source whenever it cannot proceed with reading the document. Are you sure nothing is overwriting the file? Because that error signifies that the pickled byte-string is incomplete, like if you forgot a closing `"` or `}` in a JSON document. – Edward Minnix Jun 18 '18 at 13:34
  • 3
    It should not, but there is a slight risk than another process has accessed the file to read it when it was being written by another process... However proba was so low that I didn't implemented the safety, might be the reason... – Mathieu Jun 18 '18 at 13:42
  • I just read in a [comment](https://stackoverflow.com/a/19191885/9059420) that `_pickle` lacks the `HIGHEST_PROTOCOL` constant. Maybe try without this "-1" or just straightly import `pickle` in case you're using python 3. – Darkonaut Jun 18 '18 at 14:00

0 Answers0