1

I'm trying to pickle.dump a list of lists. I suppose it's quite large with 2962694 elements, (each a list of ~2000 elements) and total size of 230MB.

I have no problems working with this list until I want to pickle it. It keeps raising:

OSError: [Errno 28] No space left on device

I tried using all the protocols, and I've tried _pickle/cPickle to no success. Each time a pickle file is created, but it's incomplete and loading it raises:

EOFError: Ran out of input

I'm using python 3.7 with Windows VM, I have TBs of space on the destination drive, I have 2.7GB on C:. I know that this error can be caused by limitations other than the hard-drive space, but I could've sworn I dumped much larger pickles - what would be other potential reasons for this error?

yassem
  • 41
  • 6
  • Check [this](https://stackoverflow.com/questions/6998083/python-causing-ioerror-errno-28-no-space-left-on-device-results-32766-h) perhaps – Devesh Kumar Singh Jun 19 '19 at 20:21
  • Can you try to dump it on another computer to see if your estimation of the size is accurate? – Imperishable Night Jun 19 '19 at 20:28
  • You described your list as "2962694 elements, each a list of ~2000 elements)". This cannot be right, can it? Did you mean that it is a list of lists with 2962694 elements on the second level? – Imperishable Night Jun 19 '19 at 20:33
  • 1
    Check the amount of free space on the drive with the temp folder on it within the VM, because it may be being used as part of the pickling process. – martineau Jun 19 '19 at 20:33
  • are you using a 32bit or 64bit python distribution? – jeschwar Jun 19 '19 at 21:32
  • @DeveshKumarSingh I started with this but none of the solutions work, or at least I cannot make them work. I'm definitely not exceeding the file count in the dir, I verified that I do have at least 2.7GB of space on C: where the temp folder is located, the '%env JOBLIB_TEMP_FOLDER=/tmp' didn't help either. The one thing I didn't try was refragmenting the drive because it seems I don't have permission. – yassem Jun 20 '19 at 10:45
  • @ImperishableNight I'll try moving it to another computer today (I wanted to avoid that as this is going to be quite cumbersome) - the list is indeed 2962694 long, but I mistook it for another object - here the sublists are much shorted (lower tens). – yassem Jun 20 '19 at 10:48
  • @jeschwar I'm using the 64bit version, the same as the os – yassem Jun 20 '19 at 10:49
  • I forgot to mention that I'm doing this through Jupyter, but I don't suppose this can be the reason? I'll try to reproduce the problem using PyCharm soon. – yassem Jun 20 '19 at 10:50
  • Oooh, ok. So now this happened - I cannot start a PyCharm project due to OSError: [Errno 28] No space left on device error - this definitely didn't happen before... – yassem Jun 20 '19 at 10:53
  • I even managed to clean up C: a little bit, up to 4GB (according to the drive's properties and running `wmic logicaldisk get size,freespace,caption` in command prompt) - I think I'll just try to contact the IT dpt about this. – yassem Jun 20 '19 at 10:59
  • Ok, turns out that the disk I assumed was for storing data which shows as having 16.4TB of free space is actually extremely limited per user. The problem is that there is apparently no way of seeing how much space is left on it for me. Anyway, thank you all for trying to help! – yassem Jun 20 '19 at 11:14

1 Answers1

0

So the actual problem was rather trivial - turns out that the system administrator imposed per user space limits on the disk I was writing to, which I couldn't see. I'm leaving the question here as it might serve as a lesson that determining the actual amount of available space is trickier than it seems.

yassem
  • 41
  • 6