8

Is there a library for creating zip files (the zip file format not gzip or any other compression format) on-the-fly (so I can start sending the file while it is compressing) for very large files (4 Gb and above).

The compression ratio does not matter much (mostly media files).

The library has to have a c-interface and work on Debian and OSX.

Daniel O
  • 4,607
  • 6
  • 44
  • 52

3 Answers3

4

libarchive supports any format you want, on the fly and even in-memory files.

wormsparty
  • 2,481
  • 19
  • 31
3

zlib supports compressing by chunks. you should be able to start sending a small chunk right after compressing it, while the library is still compressing the next chunk. (see this example)

(unfortunately, the file table is stored at the end of the zip file, so the file will be unusable until it is complete on the receiver side)

Adrien Plisson
  • 22,486
  • 6
  • 42
  • 73
  • I was under the impression that zlib cannot handle zip-files? See http://www.zlib.net/zlib_faq.html#faq11 – Daniel O Aug 30 '11 at 10:58
  • 1
    Not true about zip files, the directory is added at the end to simplify random access, but the file entries themselves are written sequentially. The directory at the end is only for convenience. – Patrick Schlüter Aug 30 '11 at 11:38
  • @DanielW: that same FAQ link points to `contrib/minizip` which does create zip files using zlib, so it obviously can. – Hasturkun Aug 30 '11 at 16:28
  • @Hasturkun no. The Minizip project have a zlib dependency and can read zip files. That doesn't automatically mean that Minizip has support for on-the-fly compression nor does it mean that zlib supports zip-files. If the answer was phrased as: You can create this zip library yourself easily by using zlib or you can check if Minizip has support for on-the-fly compression I would be more inclined to mark it as accepted. – Daniel O Aug 31 '11 at 09:48
  • 1
    well, technically, pointing you toward a library then telling you to check for yourself is not an answer to your question... that's why i will not correct this answer. – Adrien Plisson Aug 31 '11 at 09:51
  • also i had a look at the LZMA library hosted on [7-zip.org](http://7-zip.org/sdk.html): it supports decompressing in chunks but not compressing... – Adrien Plisson Aug 31 '11 at 09:58
  • now there is another answer which seems more valid than mine, i am considering deleting this answer... – Adrien Plisson Aug 31 '11 at 11:21
2

While this question is old and already answered I will note a new potential solution for those that find this.

I needed something very similar, a portable and very small library that created ZIP archives in a streaming fashion in C. Not finding anything that fit the bill I created one that uses zlib, available here:

https://github.com/CTrabant/fdzipstream

That code only depends on zlib and essentially provides a simple interface to creating ZIP archives. Most importantly (for me) the output can be streamed to a pipe, socket, whatever as the output stream does not need to be seek-able. The code is very small, a single source file and a header file. Works on OSX and Linux and probably elsewhere. Hope it helps someone beyond just me...

terse
  • 327
  • 2
  • 10