How to use the 7z DLL to compress and append many small chunks of data to a file

Question

I would like to use the 7z DLL to append small amounts of data to one compressed file. At the moment my best guess would be uncompress the 7z file, append the data, and recompress it. Obviously, this is not a good solution performance wise, if the size of the 7z file becomes large (say 1 gb) and I want do save a new chunk every second. How could I do this in a better way?

I could use any compression format supported by the 7z DLL.

@Jakob Bowyer: Please post your answer as an answer so we can upvote it and comment on it properly. — S.Lott, May 17 '11 at 13:53
Does it have to be a 7z file, or would any kind of compression work? Python also has the built-in `zipfile` library to do similar things with ZIP files. — Ben Hoyt, May 17 '11 at 14:42

CharlesB · Answer 1 · 2011-05-17T18:42:31.193

2

Have a look at the Python LZMA bindings (LZMA is the compression algorithm name of 7z), you should do what you want without ctypes stuff.

EDIT

To be confirmed, but a quick look at py7zlib.py shows only support for reading 7z files, not writing. However in the src dir there's a pylzma_compressfile.c, so maybe there's something to do.

EDIT 2

The pylzma.compressfile function seems to be there, so fine.

edited May 17 '11 at 18:42

answered May 17 '11 at 15:00

CharlesB

86,532
28
194
218

1

So how can I compress many small chunks of data to a single file without recompressing the entire file each time? – David May 22 '11 at 22:42

score 0 · Answer 2 · edited May 23 '17 at 12:04

This is NOT MY ANSWER.

How can I use a DLL file from Python?

I think ctypes is the way to go.

The following example of ctypes is from actual code I've written (in Python 2.5). This has been, by far, the easiest way I've found for doing what you ask.

import ctypes

# Load DLL into memory.

hllDll = ctypes.WinDLL ("c:\\PComm\\ehlapi32.dll")

# Set up prototype and parameters for the desired function call.
# HLLAPI

hllApiProto = ctypes.WINFUNCTYPE (ctypes.c_int,ctypes.c_void_p,
    ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p)
hllApiParams = (1, "p1", 0), (1, "p2", 0), (1, "p3",0), (1, "p4",0),

# Actually map the call ("HLLAPI(...)") to a Python name.

hllApi = hllApiProto (("HLLAPI", hllDll), hllApiParams)

# This is how you can actually call the DLL function.
# Set up the variables and call the Python name with them.

p1 = ctypes.c_int (1)
p2 = ctypes.c_char_p (sessionVar)
p3 = ctypes.c_int (1)
p4 = ctypes.c_int (0)
hllApi (ctypes.byref (p1), p2, ctypes.byref (p3), ctypes.byref (p4))

The ctypes stuff has all the C-type data types (int, char, short, void*, ...) and can pass by value or reference. It can also return specific data types although my example doesn't do that (the HLL API returns values by modifying a variable passed by reference).

How to use the 7z DLL to compress and append many small chunks of data to a file

2 Answers2