How to use LWJGL's LZ4 bindings to compress and decompress

Question

I am attempting to create two helper methods, one to compress and one to decompress. These methods should use LWJGL's LZ4 bindings to accomplish the compression/decompression.

They mostly work already but the byte array that get's returned after decompressing has trailing zeros.

import static org.lwjgl.util.lz4.LZ4.LZ4_compressBound;
import static org.lwjgl.util.lz4.LZ4.LZ4_compress_default;
import static org.lwjgl.util.lz4.LZ4.LZ4_decompress_safe;

import java.nio.ByteBuffer;

public static byte[] compress(byte[] data) {
    ByteBuffer buffer = BufferUtils.createByteBuffer(data.length);
    for (int i = 0; i < data.length; i++) {
        buffer.put(data[i]);
    }
    buffer.flip();

    ByteBuffer destination = BufferUtils.createByteBuffer(LZ4_compressBound(buffer.remaining()));
    destination.clear();

    LZ4_compress_default(buffer, destination);

    return getByteArrayFromByteBuffer(destination);
}

public static byte[] decompress(byte[] data) {
    ByteBuffer buffer = BufferUtils.createByteBuffer(data.length);
    for (int i = 0; i < data.length; i++) {
        buffer.put(data[i]);
    }
    buffer.flip();

    ByteBuffer destination = BufferUtils.createByteBuffer(LZ4_compressBound(buffer.remaining()));
    destination.clear();

    LZ4_decompress_safe(buffer, destination);

    return getByteArrayFromByteBuffer(destination);
}

public static byte[] getByteArrayFromByteBuffer(ByteBuffer byteBuffer) {
    byte[] bytesArray = new byte[byteBuffer.remaining()];
    byteBuffer.get(bytesArray, 0, bytesArray.length);
    return bytesArray;
}

The result after running compress(decompress(SOME DATA)), is: [SOME_DATA, 0, 0, 0].

The first half of the data is correct but for some reason extra zeros are added. This is probably due to the ByteBuffers not being correctly setup, but I am unsure. Either way how can I remove these extra zeros?

Also unfortunately this is the only link I could find that shows an example on how this is supposed to work.

score 1 · Accepted Answer · answered Jul 01 '19 at 17:52

First of all, during compression you are currently using the result of LZ4_compressBound(...) to define the initial capacity of your destination buffer. While this is correct, you have to keep in mind that this is only the Worst-Case size of the compressed data. The actual amount of bytes written by LZ4_compress_default(...) is the return value of said function and likely less than the bound. Thus you need to trim your destination buffer to match that size.

int compressedSize = LZ4_compress_default(buffer, destination);
destination.limit(compressedSize);
destination = destination.slice();

Additionally you have a similar bug in your decompression: You can not use LZ4_compressBound(...) to calculate the capacity of your destination buffer. Instead, you need to store the size of the decompressed data elsewhere. (In theory you could use compressedSize * 255 but this is extremely impractial since that compression ratio is almost never achieved and thus leads to a huge amount of wasted resources. See here.) And again you need to respect the return value of LZ4_decompress_safe which is the actual size of the decompressed data.

How to use LWJGL's LZ4 bindings to compress and decompress

1 Answers1