Writing ZRLE Encoding for VNC Server

Question

I've been writing a VNC Server using rust as an academic exercise. I've successfully implemented RAW and Hextile encoding. I've laid out blocks for ZLib are ZRLE. But unfortunately, ZLIB compression sent by my server does not inflate in any VNC Client.

I've done a lot of research and read the source codes of LibVNCServer, UltraVNC and other open-source VNC Servers. My code is almost same, it's the same API, but the compressed data sent by my server fails to inflate with code -3 (msg: invalid stored block lengths). I went through the RFB Protocol Documentation while implementing it. I've implemented it to the best of my understanding, I'll attach the source code here and since my project is open-source the full code is at https://github.com/candiedoperation/spifyrfb-v2. The encoding source code is at https://github.com/candiedoperation/spifyrfb-v2/tree/master/protocol/src/server

use std::{mem, ptr};

#[derive(Debug)]
pub struct ZlibPixelData {
    pub pixel_data_len: u32,
    pub pixel_data: Vec<u8>
}

pub fn deflate(pixel_data: Vec<u8>) -> ZlibPixelData {
    let max_compressed = pixel_data.len() + ((pixel_data.len() + 99) / 100) + 12;
    let mut next_in: Vec<u8> = pixel_data.clone();
    let mut next_out: Vec<u8> = vec![0; max_compressed];

    unsafe {
        /* Define z_stream struct */
        let mut zlib_stream = libz_sys::z_stream {
            next_in: next_in.as_mut_ptr(),
            avail_in: next_in.len() as u32,
            total_in: 0,
            next_out: next_out.as_mut_ptr(),
            avail_out: max_compressed as u32,
            total_out: 0,
            msg: ptr::null::<u8>() as _,
            state: ptr::null::<u8>() as _,
            zalloc: mem::transmute(ptr::null::<u8>()),
            zfree: mem::transmute(ptr::null::<u8>()),
            opaque: ptr::null::<u8>() as _,
            data_type: libz_sys::Z_BINARY,
            adler: 0,
            reserved: 0,
        };
        
        /* Call deflateInit2_ */
        let deflate_init_status = libz_sys::deflateInit2_(
            &mut zlib_stream,
            6, /* Set Compress Level 6 (0-9, None-Max) */
            libz_sys::Z_DEFLATED,
            15, /* Range: 8-15 (Min-Max Memory) */
            8,
            libz_sys::Z_DEFAULT_STRATEGY,
            libz_sys::zlibVersion(),
            mem::size_of::<libz_sys::z_stream>() as i32,
        );

        if deflate_init_status != libz_sys::Z_OK {
            println!("ZLIB: DeflateInit2_() failed. Status: {}", deflate_init_status);
            return ZlibPixelData { 
                pixel_data_len: pixel_data.len() as u32, 
                pixel_data
            };
        }

        let deflate_status = libz_sys::deflate(
            &mut zlib_stream,
            libz_sys::Z_FULL_FLUSH
        );

        if deflate_status != libz_sys::Z_OK {
            println!("ZLIB: Deflate() failed. Status: {}", deflate_status);
            return ZlibPixelData { 
                pixel_data_len: pixel_data.len() as u32, 
                pixel_data
            };
        }

        println!("ZLIB: Compressed: {} bits to {} bits", zlib_stream.total_in, zlib_stream.total_out);
        ZlibPixelData { 
            pixel_data_len: zlib_stream.total_out as u32, 
            pixel_data: (&next_out[..zlib_stream.total_out as usize]).to_vec()
        }
    }
}

pub fn get_pixel_data(pixel_data: Vec<u8>) -> ZlibPixelData {
    deflate(pixel_data)
}

Okay, so I found out a solution. I'll be documenting and posting it as an answer. But, other answers are welcome! :-) — Atheesh Thirumalairajan, Jun 10 '23 at 08:56

score 0 · Accepted Answer · answered Jun 10 '23 at 11:30

According to the Remote Framebuffer Protocol specification for ZRLE (and ZLib based encodings)

A single zlib “stream” object is used for a given RFB connection, so that zlib rectangles must be encoded and decoded strictly in order. Read the documentation here

I don’t know the reason for it (maybe the ZLib inflation algorithm uses some header which identifies if data was deflated by the same ZLib stream in the previous data block). Another reference which hints on this is a part of the LibVNCServer source code.

Note that it is not possible to switch zlib parameters based on the results of the compression pass. The reason is that we rely on the compressor and decompressor states being in sync. Compressing and then discarding the results would cause lose of synchronization. Read the source here

And, some references from a ZLib encoding (zlibhex.c) written by Tridia Corporation and AT&T Laboratories Cambridge hint on the fact the the stream should be initialized, the first time, if necessary. Read the Source here

So, the solution was creating a ZLib stream for every client and storing it persistently. Once the client requested a FrameBufferUpdateRequest the ZLib stream that was created for the client (and initialized using DeflateInit2()) is sent to the Deflate() function for compression. The compression succeeds, the variable that holds the ZLib stream persistently is updated with the new ZLib stream created by the Deflate() function. And finally, the compressed ZLib data is sent to the VNC client.

However, this answer does not have a reason to why the same stream needs to be used for the client and re-constructing the stream (calling DeflateInit2_()) or using a different ZLib stream object (a stream object which was not used for sending the first frame) for encoding frames causes the VNC client to fail. So, if someone has an answer for this, it’ll get upvoted, for sure.

Writing ZRLE Encoding for VNC Server

1 Answers1