0

I'm trying to unzip a 8gb csv file with a machine 8gb RAM. This is my implementation, using python 3.6:

def extract_to(zippath, dest):
with zipfile.ZipFile(zippath, mode='r') as z:
    for member in z.namelist():
        filename = Path(member).basename()
        # skip directories
        if not filename:
            continue

        source = z.open(member)
        target = open(dest / filename, "wb")
        with source, target:
            shutil.copyfileobj(source, target)

Unfortunately, it produces the following error:

Traceback (most recent call last):
  File "download_data.py", line 31, in extract_to
    shutil.copyfileobj(source, target)
  File "/home/user/anaconda3/lib/python3.6/shutil.py", line 82, in copyfileobj
    fdst.write(buf)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "download_data.py", line 56, in <module>
    main()
  File "download_data.py", line 49, in main
    download_and_extract_zip(train, RAW_DATA_DIR)
  File "download_data.py", line 37, in download_and_extract_zip
    extract_to(tmppath, dest)
  File "download_data.py", line 31, in extract_to
    shutil.copyfileobj(source, target)
OSError: [Errno 28] No space left on device

How can I handle this situation? Thanks.

Giovanni Barbarani
  • 239
  • 1
  • 11
  • 4
    This isn't about RAM; your PC is out of disk space. Delete some stuff I guess? – Aran-Fey Mar 19 '18 at 00:18
  • 1
    If the zip file contains several files, you could first find out which they are and then uncompress only one at a time. – Arndt Jonasson Mar 19 '18 at 00:20
  • @Aran-Fey I do not think it's a problem with hard disk space. It's a completely new vm with 100gb of hard disk. – Giovanni Barbarani Mar 19 '18 at 00:27
  • @ArndtJonasson unfortunately it is a single 8gb cvs file. – Giovanni Barbarani Mar 19 '18 at 00:29
  • https://stackoverflow.com/questions/6998083/python-causing-ioerror-errno-28-no-space-left-on-device-results-32766-h in the first answer the author shows how this error can have different causes. But I can't figure out a solution for this case. – Giovanni Barbarani Mar 19 '18 at 00:31
  • 1
    Try downloading and unzipping the file by hand. If you get the same error, you are indeed out of disk space. – DYZ Mar 19 '18 at 00:45
  • 1
    `OSError: [Errno 28] No space left on device` doesnt sound like a RAM error to me. Also the fact that the error occurs during a call to `shutil.copyfileobj`. – Paul Rooney Mar 19 '18 at 01:13
  • 1
    You could try the command `python -m zipfile -e yourzip.zip target-dir/` to see how python gets along with decompressing your file by itself. – Paul Rooney Mar 19 '18 at 01:20
  • 1
    It should be possible to unzip to stdout, so that you can read successive lines from the file, without having to have the whole file on disk or in memory. Maybe the zipfile module has such a function (I haven't looked). – Arndt Jonasson Mar 19 '18 at 09:36
  • You guys are right. By mistake I had created a VM with only 10GB of disk. Thanks. – Giovanni Barbarani Mar 19 '18 at 12:59
  • I ran into this on AWS using /efs/ virtual disk space. The container had 256 GIGS of memory and 100GB of EFS, and the zipfile was only 16GB. IT appears to have something to do with the way the drive is formatted and inodes and blocks and such. https://serverfault.com/questions/482173/is-there-any-other-reason-for-no-space-left-on-device Haven't fixed it yet, but I am closing in... – Marc Maxmeister Jun 24 '21 at 16:23

0 Answers0