Unzip buffer with large data length is crashing

Question

This is the function I am using to unzip buffer.

string unzipBuffer(size_t decryptedLength, unsigned char * decryptedData)
{
    z_stream stream;
    stream.zalloc = Z_NULL;
    stream.zfree = Z_NULL;
    stream.avail_in = decryptedLength;
    stream.next_in = (Bytef *)decryptedData;
    stream.total_out = 0;
    stream.avail_out = 0;
    size_t dataLength = decryptedLength* 1.5;
    char c[dataLength];

    if (inflateInit2(&stream, 47) == Z_OK)
    {
        int status = Z_OK;
        while (status == Z_OK)
        {
            if (stream.total_out >= dataLength)
            {
                dataLength += decryptedLength * 0.5;
            }

            stream.next_out = (Bytef *)c + stream.total_out;

            stream.avail_out = (uint)(dataLength - stream.total_out);

            status = inflate (&stream, Z_SYNC_FLUSH);

        }
        if (inflateEnd(&stream) == Z_OK)
        {
            if (status == Z_STREAM_END)
            {
                dataLength = stream.total_out;
            }
        }
    }
    std::string decryptedContentStr(c, c + dataLength);
    return decryptedContentStr;
}

And it was working fine until today when I realized that it crashes with large data buffer (Ex: decryptedLength: 342792) on this line:

status = inflate (&stream, Z_SYNC_FLUSH);

after one or two iterations. Can anyone help me please?

Maybe you should tell exactly how it crashed. BTW - could this be a stack overflow issue? `char c[dataLength];` is approx. 600.000 chars. — Support Ukraine, Aug 13 '15 at 15:07

score 2 · Accepted Answer · edited May 23 '17 at 11:58

2

If your code generally works correctly, but fails for large data sets, then this could be due to a stack overflow as indicated by @StillLearning in his comment.

A usual (default) stack size is 1 MB. When your decryptedLength is 342,792, then you try to allocate 514,188 byte in the following line:

char c[dataLength];

Together with other allocations in your code (and finally in the inflate() function), this might already be too much. To overcome this problem, you should allocate the memory dynamically:

char* c = new char[dataLength];

If you so this, then please do not forget to release the allocated memory at the end of your unzipBuffer() function:

delete[] c;

If you forget to delete the allocated memory, then you will have a memory leak.

In case this doesn't (fully) solve your problem, you should do it anyway, because for even larger data sets your code will break for sure due to the limited size of the stack.

In case you need to "reallocate" your dynamically allocated buffer in your while() loop, then please take a look at this Q&A. Basically you need to use a combination of new, std::copy, and delete[]. However, it would be more appropriate if your exchange your char array with a std::vector<char> or even std::vector<Bytef>. Then you would be able enlarge your buffer easily by using the resize() function. You can directly access the buffer of a vector by using &my_vector[0] in order to assign it to stream.next_out.

edited May 23 '17 at 11:58

Community

1
1

answered Aug 13 '15 at 16:43

honk

9,137
11
75
83

thanks for the help, but the problem persists. And thanks for the advice, I will keep allocating the memory dynamically. – Grace Aug 14 '15 at 06:07
You are welcome! Please also take a look at the answer by @MarkAdler. He raises a point that also made me suspicious. I didn't include it, because I don't exactly know how the zlib works. Just try it with 3 instead of 1.5 times. If it works, then you need to dynamically reallocate your buffer in order to enlarge it. – honk Aug 14 '15 at 06:40
@Grace: I added some explanation on the "reallocation" issue to my answer. – honk Aug 14 '15 at 07:09
thanks for the great help, I'll change my code to meet your explanation and let you know once done – Grace Aug 14 '15 at 07:24
Works very well. Thanks :D – Grace Aug 14 '15 at 07:44
I'm glad to have been of help :) – honk Aug 14 '15 at 08:01

score 1 · Answer 2 · answered Aug 13 '15 at 23:55

c is not going to get bigger just because you increase datalength. You are probably overwriting past the end of c because your initial guess of 1.5 times the compressed size was wrong, causing the fault.

(It might be a stack overflow as suggested in another answer here, but I think that 8 MB stack allocations are common nowadays.)

Unzip buffer with large data length is crashing

2 Answers2