Is there any reason GZipStream.Read would return less than the number of requested bytes?

Question

Return Value:
The total number of bytes read into the buffer. This can be less than the number of bytes requested if that many bytes are not currently available, or zero (0) if the end of the stream has been reached.

But why would the bytes "not be available" when reading from disk?

Let me clarify a bit:

I'm reading from disk (underlying type is FileStream)
There are at least N bytes left to be read (before EOF)
I request to read N bytes

Will the return value/number of bytes read ever be less than N in this scenario?

Have you seen this answer? http://stackoverflow.com/questions/5075924/what-does-filestream-read-return-value-mean-how-to-read-data-in-chunks-and-proc — Lloyd, Feb 17 '12 at 04:55
Because you reached the end of the stream before you filled the buffer. — phoog, Feb 17 '12 at 05:35
I was brought back to this question by a comment to my answer. It occurs to me that the answer to "why would the bytes 'not be available' when reading from disk?" might be "because the disk is busy doing something else for another process." — phoog, Jul 30 '14 at 17:28
This has just caught me out. For me, Read() has always returned the number of bytes that I've asked for (yes I wrote my code around this dangerous assumption), but after porting my app to .Net 6 Read() is typically returning fewer than requested, so I guess the GZipStream implementation has changed. Your Stream extension below did the trick thanks! — Andrew Stephens, Sep 15 '22 at 11:07
I encountered this very issue when switching from .Net Framework 4.6.2 to Net 6 while reading the same file, per MS, this is a breaking change. https://learn.microsoft.com/en-us/dotnet/core/compatibility/core-libraries/6.0/partial-byte-reads-in-streams The problem is there is no workaround - the "offset" parameter doesn't work at all. — Tom Charles Zhang, Oct 31 '22 at 20:18

score 3 · Accepted Answer · answered Feb 17 '12 at 15:56

3

In answer to your edited question:

Will the return value/number of bytes read ever be less than N in this scenario?

I think you need to ask a hardware expert, and I suppose this isn't the right forum to attract the attention of such a person.

Disclaimer: I'm not a hardware expert, and I don't play one on TV. This is just speculation:

I think that when you're reading from disk, the only reason you'd get fewer bytes than you request is because the stream has run out of bytes to give you. However, it's conceivable that you might have a situation similar to a network stream, where your program is reading bytes faster than the hardware can provide them. In that case, the Read method would presumably populate the buffer only partially and then return.

Obviously, the answer to the question depends on whether such a situation could occur. I think the answer is "no". I have certainly never seen a counterexample. But it would be a mistake to write code depending on that assumption.

Consider: even if you could examime the specifications of all the hardware your code will run on, and prove that the buffer will always be completely filled until the end of the stream is reached, there's no saying what new disk drive somebody might install on the machine in the future, that might behave differently. It's much simpler just to treat all streams the same, and undertake the modest amount of work required to handle the possibility that the buffer comes back incompletely filled.

answered Feb 17 '12 at 15:56

phoog

42,068
6
79
117

Fair enough. I'll write an extension method to do what I want. Seems odd that there isn't a built-in function for it already. – mpen Feb 17 '12 at 21:48
FileStream can return less bytes that asked for even if the stream/file has not ended. See: http://msdn.microsoft.com/en-us/library/system.io.filestream.read.aspx "An implementation is free to return fewer bytes than requested **even if the end of the stream has not been reached**" – data Jul 07 '14 at 11:50
@data I understand that; that's why I wrote "it would be a mistake to write code depending on that assumption." But even though an implementation is free to do so, it might in fact be designed so that it never does. – phoog Jul 30 '14 at 17:26
@phoog what does that change? Even if in some version, some implementation does not return less that requested, a future version might. The documentation clearly says to not depend on all the requested bytes being returned. What are we arguing about? – data Jul 31 '14 at 19:09
@data I don't think it changes much, but the question seems to be asking about implementation details (that is, the possibility that when reading from disk, FileStream might never actually take advantage of its option to return fewer bytes than requested). The question didn't seem to be about the method contract but rather about implementation-specific behavior. If the question is about the behavior of a specific implementation, then pointing out how the implementation is **allowed** to behave doesn't help. That's why my answer is basically "I don't know, so assume that the answer is yes." – phoog Aug 18 '14 at 21:27

score 2 · Answer 2 · answered Jul 30 '14 at 23:38

My solution:

public static class StreamExt
{
    public static void ReadBytes(this Stream stream, byte[] buffer, int offset, int count)
    {
        int totalBytesRead = 0;
        while (totalBytesRead < count)
        {
            int bytesRead = stream.Read(buffer, offset + totalBytesRead, count - totalBytesRead);
            if (bytesRead == 0) throw new IOException("Premature end of stream");
            totalBytesRead += bytesRead;
        } 
    }
}

Using this method should safely read in all the bytes you requested.

M.Babcock · Answer 3 · 2012-02-17T13:25:48.527

1

Stream.Read requires that you pre-allocate the space to read into, and if you allocate more space then can be read from the stream, then the return value is its way of telling you how much of it has been used.

Consider the following:

If you allocated a buffer of say 4096 bytes and the EOF is reached at 2046, then the return value would only be 2046. This allows you to know how much of your buffer is full on return.

edited Feb 17 '12 at 13:25

answered Feb 17 '12 at 04:51

M.Babcock

18,753
6
54
84

Or am I thinking too simplistically for your problem? – M.Babcock Feb 17 '12 at 05:01
Yes, the number of bytes read might be less than the size of my buffer, but I'm asking about when it would be less than the number of bytes I requested. See update. **Edit:** Took me awhile to notice the extra 0 in the buffer size in your example; confused me for a bit. – mpen Feb 17 '12 at 07:20
@Mark - Changed the buffer size in my answer to something more distinct. After seeing your edit, are you experiencing a problem currently with this or are you trying to avoid the issue from happening if it *is* possible? If the latter, it seems like a valid concern but if it was a common problem then I think we'd be seeing more questions about it. – M.Babcock Feb 17 '12 at 16:43
@M.Barbock: Trying to prevent it from happening. Seems funny if I have to write a loop every time just to make sure I get the bytes I requested. I can't really do anything until I get *all* the bytes. – mpen Feb 17 '12 at 21:45

score 1 · Answer 4 · edited May 23 '17 at 10:34

1

Compression streams are wrapper stream - so behavior depends on behavior of underlying stream. As result it has the same restriction as generic stream "can return less bytes than requested".

As for potential reasons (see also provided by Lloyd)

EOF reached on last read
No data in network stream yet
Custom stram decided to return data in fixed size chunks (perfectly ok).

edited May 23 '17 at 10:34

Community

1
1

answered Feb 17 '12 at 06:40

Alexei Levenkov

98,904
14
127
179

Is there any reason GZipStream.Read would return less than the number of requested bytes?

4 Answers4