How to zip huge plain files (+20GB) with python's ziplib file

Question

I get DeprecationWarning: struct integer overflow masking is deprecated error in z.write when file arrives to 4GB.

My code:

def compressOutputFile(outputFileName, outputPath=UXConfig.myPath):
  os.chdir(outputPath)
  z= zipfile.ZipFile(outputFileName+'.zip', 'wb',zipfile.ZIP_DEFLATED,allowZip64=True)
  UXUtils.log('Writting file')
  z.write(outputFileName)
  UXUtils.log('Writting finished')
  z.close()
  tempFiles.append(outputPath+outputFileName)

The file isn't corrupted because I can open and see the lines.

score 1 · Answer 1 · edited May 23 '17 at 12:25

1

I never used the zipfile module but I used a with statement and tried it against an 8GB file and it worked (from 8GB to 44MB :o):

def zipItUp(file):
    with zipfile.ZipFile('zipped.zip', 'w', compression=zipfile.ZIP_DEFLATED, allowZip64=True) as zipped:
        zipped.write(file)

See if this does anything. If not, I think the only other solution I can think of is partitioning your file up and then the zip on each chunk.

EDIT: If you're using Python 2.6, then consider including the contextlib module to handle the exit error as referenced in this topic.

Here would be the revised code:

import contextlib

def zipItUp(file):
    with contextlib.closing(zipfile.ZipFile('zipped.zip', 'w', compression=zipfile.ZIP_DEFLATED, allowZip64=True)) as zipped:
        zipped.write(file)

See if this works. Cheers!

edited May 23 '17 at 12:25

Community

1
1

answered Apr 27 '16 at 18:20

Scratch'N'Purr

9,959
2
35
51

I'm using Python 2.6 and it raises "zipfile instance has no attribute '__exit__'" – Laura Abad Avilés Apr 28 '16 at 06:29
@LauraAbadAvilés I just added an edit for the 2.6 implementation. Hope it helps. – Scratch'N'Purr Apr 28 '16 at 12:44
still failing. It always generate a file with 4.926.126.365 bytes that seems correct, but when I try to extract file tells me sumcontrol error. Maybe ZIpFile can't create zip files with that size. – Laura Abad Avilés Apr 28 '16 at 15:08
No progression on this problem? Does `shutil` work with large zip files? – krenerd Feb 02 '21 at 05:24

How to zip huge plain files (+20GB) with python's ziplib file

1 Answers1