1

I have a small client application designed to run on windows machines. It is written in C/C++ (more in C). I want to keep it small and therefore I would prefer not to use external libraries and stick to WinAPI on client side.

On the other hand I have a backend server which is implemented in python3. Here I'm happy to use any existing lib. Now I want to add compression layer to increase transfer speed. The problem I've encountered is that it seems that WinAPI provides only: 

// source: https://learn.microsoft.com/en-us/windows/win32/cmpapi/using-the-compression-api
XPRESS
XPRESS with Huffman encoding 
MSZIP  
LZMS

which seem to be a unique Microsoft compression algorithm implementation and I cannot find a way to decompress the data on the server side in python3.

Is there anything I am missing? I would love to hear some solutions:)

Thanks in advance

UPDATE

I've decided to use zlib https://zlib.net/ as suggested in the comment. The answer suggesting using ctypes was also very interesting, but unfortenately my backend is running on UNIX system.

As I am compiling my client part agains Multi-threaded CRT (and not DLL) I had some problems as zlib is being linked agains the Muli-threaded DLL. If anyone has enountered such issue, I have found a great and super simple solution here: https://yongweiwu.wordpress.com/2017/10/02/a-journey-of-purely-static-linking/ I will copy-paste it here:

zlib
This part requires a small change to the build script (for version 1.2.11).
I need to open win32\Makefile.msc and change all occurrences of ‘-MD’ to ‘-MT’. 
Then these commands will work:

nmake -f win32\Makefile.msc zlib.lib
Rob D
  • 71
  • 1
  • 6
  • 1
    *I want to keep it small and therefore I would prefer not to use external libraries and stick to WinAPI on client side* -- Have you tried `zlib`?.. – PaulMcKenzie May 07 '20 at 20:04
  • I don't want to use external libraries:/ – Rob D May 07 '20 at 20:18
  • 1
    `zlib` is a set of C code that implements the flate compression algorithm, the same compression used in PDF, PNG, TIFF files, and probably more. Dismissing it as an external library is short-sighted, to be honest with you. It is probably also small that the difference in size in the final executable is not even that noticeable (you can just add the 'C' source code to your project). Also [this link](https://stackoverflow.com/questions/1089662/python-inflate-and-deflate-implementations) suggests that Python supports zlib. – PaulMcKenzie May 08 '20 at 04:12

1 Answers1

0

I found this Python script, and reverse-engineered it to produce a quick library to handle WinAPI Compression/Decompression. Basically, you can just use ctypes and call the WinAPI from Python. Please keep in mind that I haven't extensively tested this, but it should give you a pretty good place to start :)

EDIT: As requested, I've included the implementations of the compress and decompress functions in case the links ever go down.

def compress(UncompressedBuffer, UncompressedBufferSize, Format, Engine):
    CompressedBuffer = (ctypes.c_ubyte * UncompressedBufferSize)()
    CompressionFormatAndEngine = ctypes.c_uint16(Format.value | Engine.value)

    CompressBufferWorkSpaceSize = ctypes.c_uint32()
    CompressFragmentWorkSpaceSize = ctypes.c_uint32()
    WorkSpace = (CompressFragmentWorkSpaceSize.value * ctypes.c_ubyte)()
    FinalCompressedSize = ctypes.c_uint32()

    WinDLLCall(RtlGetCompressionWorkSpaceSize,
               CompressionFormatAndEngine,
               ctypes.byref(CompressBufferWorkSpaceSize),
               ctypes.byref(CompressFragmentWorkSpaceSize))

    WinDLLCall(RtlCompressBuffer,
              CompressionFormatAndEngine,
              ctypes.byref(UncompressedBuffer),
              ctypes.c_uint32(UncompressedBufferSize),
              ctypes.byref(CompressedBuffer),
              ctypes.c_uint32(UncompressedBufferSize),
              UncompressedChunkSize,
              ctypes.byref(FinalCompressedSize),
              ctypes.byref(WorkSpace))

    return CompressedBuffer, FinalCompressedSize

def decompress(CompressedBuffer, CompressedBufferSize, UncompressedBufferSize, Format):
    UncompressedBuffer = (ctypes.c_ubyte * UncompressedBufferSize)()
    FinalUncompressedSize = ctypes.c_uint32()

    WinDLLCall(RtlDecompressBuffer,
               Format,
               ctypes.byref(UncompressedBuffer),
               ctypes.c_uint32(UncompressedBufferSize),
               ctypes.byref(CompressedBuffer),
               ctypes.c_uint32(CompressedBufferSize),
               ctypes.byref(FinalUncompressedSize))

    return UncompressedBuffer, FinalUncompressedSize

The WinAPI functions are loaded with the following code:

import ctypes

RtlDecompressBuffer = ctypes.windll.ntdll.RtlDecompressBuffer
RtlCompressBuffer = ctypes.windll.ntdll.RtlCompressBuffer
RtlGetCompressionWorkSpaceSize = ctypes.windll.ntdll.RtlGetCompressionWorkSpaceSize

Here are some other useful resources if you end up tinkering around:

Cristian Bicheru
  • 59
  • 4
  • This boils down to a link-only answer. Once you omit the links, there's no useful information left. Please see the paragraph *"Provide context for links"* under [How do I write a good answer?](https://stackoverflow.com/help/how-to-answer) and provide the required changes. – IInspectable May 08 '20 at 06:13
  • This is actually very good answert:) I havn't considered ctypes, but I will. Still hopying for some other method as my backend is running on unix and I would have to copy the dll from windows. – Rob D May 08 '20 at 06:31
  • 1
    @rob You cannot simply copy a kernel module from Windows and expect it to run on anything but Windows. This won't work on a technical level, and it has legal implications, too. – IInspectable May 08 '20 at 08:07
  • @RobD As stated by @IInspectable copying the DLLs might be a little wonky, but I did find this open source library called [WIMLIB](https://wimlib.net/) which contains implementations of the compression algorithms. The API looks pretty similar to the WinAPI, if you do end up using `ctypes` you can probably get it working with some slight tweaks. – Cristian Bicheru May 08 '20 at 11:09