Download a file with HttpClient but defer writing

Question

I can download byte[]'s and write them to a file no problem, much link this link does. But when I make a simple alteration, saving the bytes[] and readCount to a List and then foreach over the list, I get a corrupt file.

Below is partial code that shows what I mean. There are two commented paths: one stores the bytes/counts (and order) and then writes when the reading is complete. The other writes in line with the each read.

Because of the order field, I have verified that the foreach is writing the bytes in the correct order but the partial code listing doesn't explicitly use it.

    private class ByteHolder
    {
        public byte[] Buffer { get; set; } = new byte[8192];
        public int BytesRead { get; set; }
        public int Order { get; set; }
    }

    var byteList = new List<ByteHolder>();

    using (var fileStream = new FileStream(_fileBytes.SavingPath.RuntimePath(), FileMode.Create, FileAccess.Write, FileShare.None, 8192, true))
    {
        do
        {
            var bytesRead = await contentStream.ReadAsync(buffer, 0, buffer.Length);
            if (bytesRead == 0)
            {

                //--- WRITING THE bytes and counts this way produces corrupt file -->
                var offset = 0;
                foreach (var byteHolder in byteList)
                {
                    await fileStream.WriteAsync(byteHolder.Buffer, offset, byteHolder.BytesRead).ConfigureAwait(false);
                    offset += 8192;
                }

                isMoreToRead = false;
                FileDownloadComplete?.Invoke(this, _fileBytes);
                continue;
            }

            //--- Storing them in this class and list for writing above DOES NOT WORK -->
            byteList.Add(new ByteHolder() { Buffer = buffer, BytesRead = bytesRead, Order = readCount }); 

            //--- Writing inline with the read WORKS -->
            //await fileStream.WriteAsync(buffer, 0, bytesRead);

            readCount++;
            totalBytesRead += bytesRead;

So you always writing data into file from offset 0, you need to increment offset within "foreach" loop. — Andrey Sidorenko, Jan 27 '20 at 17:27
Thanks for replying. I added the offset to the sample code above. It still generates a corrupt file. Not sure if I should offset by the buffer size or the read size, but neither works. The file I'm working with is an xlsx. (Also, the inline WriteAsync uses offset 0 and it works!?) The particular xlsx takes four reads to download. — John Mc, Jan 27 '20 at 17:59
Take a look at the documentation for WriteAsync, the offset you pass in is the offset in the *buffer*, not the file. It should always be 0; the increment is unnecessary. — Mike Zboray, Jan 27 '20 at 18:26
Where is `buffer` initialized? Here you are reusing the same buffer array on every loop which is what is causing the issue. You have to create a new buffer array every time you read. — Mike Zboray, Jan 27 '20 at 18:28
Mike Zboray, "create a new buffer every time I read" is the answer. Thanks so much! That isn't a very obvious solution. — John Mc, Jan 27 '20 at 18:43

score 0 · Answer 1 · answered Jan 27 '20 at 18:09

0

            foreach (var byteHolder in byteList)
            {
                var offset = 0;
                await fileStream.WriteAsync(byteHolder.Buffer, offset, byteHolder.BytesRead).ConfigureAwait(false);
                offset += 8192;
            }

You're always writing at offset 0, no matter which byteHolder instance you're processing at the time.

answered Jan 27 '20 at 18:09

Blindy

65,249
10
91
131

Your whole algorithm is wrong anyway, you shouldn't keep the whole thing in memory before writing it to disk, just write the pieces you get as you're getting them. – Blindy Jan 27 '20 at 18:10
I've tried this every way. Blindy noticed a problem. I fixed it. Same problem. I cannot do what you suggest because ultimately this file will be transferred over SignalR which cannot handle bytes. Before you call foul on that, I've already verified I can transfer Base64 representations over SignalR without error. The problem happens before SignalR anyway. – John Mc Jan 27 '20 at 18:15
It doesn't matter what it's transferred over, you're using an operating system construct called a socket. It doesn't even matter what you think "cannot handle bytes" means, (tcp) sockets stream bytes to you in chunks of the size you request. – Blindy Jan 27 '20 at 19:53
I wish that were true. It would have saved me a few days. Yet, when I SendAsync a List, it never triggers the .On. I change that call to send a string and the .On fires. – John Mc Jan 28 '20 at 17:15

Download a file with HttpClient but defer writing

1 Answers1