Stream from response.Content.ReadAsStreamAsync() is not readable randomly

Question

I am making a .Net Web API application in which the following code is making call to my different c# application to download file and then save it on the disk. Sometimes everything works fine and I get the file but sometimes the below code is not able to read the stream and I can see the remote connection closed exception in my other application.

public async Task<string> GetFilePathAsync(TestModel model)
{
    string filePath = string.Empty;
    var response = await cgRequestHelper.DownloadAsync(model);  

    if (response.IsSuccessStatusCode)
    {                    
        filePath = await SaveCgStreamAsync(cgResponse, serviceModel.FileName);
    }
    return filePath;
}

public async Task<HttpResponseMessage> DownloadAsync(TestModel model)
{            
    if (model == null)
        throw new ArgumentNullException("model");
    if (string.IsNullOrEmpty(model.Url))
        throw new ArgumentNullException("Url");
    if (model.Headers == null)
        throw new ArgumentNullException("Headers");

    HttpResponseMessage response;
    using (HttpClient httpClient = new HttpClient())
    {
        foreach (var header in model.Headers)
        {
            httpClient.DefaultRequestHeaders.Add(header.Key, header.Value);
        }
        response = await httpClient.GetAsync(model.Url, HttpCompletionOption.ResponseHeadersRead); 
    }
    return response;            
}        

public async Task<string> SaveCgStreamAsync(HttpResponseMessage response, string fileName)
{
    if (response == null)
        throw new ArgumentNullException("response");
    if (string.IsNullOrEmpty(fileName))
        throw new ArgumentNullException("fileName");

    var filePath = _CreateTemporaryLocation(fileName);
    Stream cgStream = null;
    Stream fileStream = null;
    try
    {
        cgStream =  await response.Content.ReadAsStreamAsync();
        fileStream = File.Open(filePath, FileMode.Create);
        await cgStream.CopyToAsync(fileStream);
    }
    catch(Exception e)
    {
        throw;
    }
    finally
    {
        if(cgStream != null)
            cgStream.Dispose();
        if(fileStream != null)
            fileStream.Dispose();
    }

    return filePath;            
}

I have set ServicePointManager.DefaultConnectionLimit = 1000 in Global.asax.cs

In my current application I am getting exception on "await cgStream.CopyToAsync(fileStream);" line when it tries to read the cgStream. The exception is "Cannot access a closed Stream." with null InnerException The other application exception is: System.Web.HttpException: The remote host closed the connection. The error code is 0x800703E3. at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect) at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush() at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)

On an average, 1 out of 10 request is failing with the above error. Since it is random and not always failing, it is very difficult to troubleshoot the problem.

The above code is written with the help from: http://www.tugberkugurlu.com/archive/efficiently-streaming-large-http-responses-with-httpclient

Any help related to this problem is appreciated!

Thanks

Answer 1

I figured out the problem in the code: The problem was that I was initializing HttpClient object in 'using' and using its response outside the using scope. This will dispose the HttpClient object, hence breaking the remote connection and that is why I am not able to stream the content. The random behavior was because we do not know when will the object gets disposed, sometimes it does not gets disposed before streaming and sometimes it does.

Answer 2

Per the documentation, Stream.CopyToAsync is, well, asynchronous. That means it isn't going to block the caller (your method), which will go on to Dispose of the stream in the finally block. You have a race condition, in other words. Switch to Stream.CopyTo and you should be good to go.

Answer 3

I will try Stream.CopyTo instead but just to let you know I was trying the below code earlier and later changed that to CopyToAsync

public async Task<string> GetFilePathAsync(TestModel model)
    {
        string filePath = string.Empty;
        var response = await cgRequestHelper.DownloadAsync(model);  

        if (response.IsSuccessStatusCode)
        {                   
            Stream cgStream = await response.Content.ReadAsStreamAsync();
            filePath = SaveCgStream(cgStream , model.FileName);
        }
        return filePath;
    }

    public string SaveCgStream(Stream cgStream, string fileName)
        {
            if (response == null)
                throw new ArgumentNullException("response");
            if (string.IsNullOrEmpty(fileName))
                throw new ArgumentNullException("fileName");

            const int CHUNK_SIZE = 40000;
            var filePath = _CreateTemporaryLocation(fileName);
            var fileStream = File.Create(filePath, CHUNK_SIZE);

            int bytesRead;
            var buffer = new byte[CHUNK_SIZE];
            bool isReadStarted = false;
            try
            {
                do
                {
                    bytesRead = cgStream.Read(buffer, 0, CHUNK_SIZE);
                    if (bytesRead > 0)
                    {
                        isReadStarted = true;
                        fileStream.Write(buffer, 0, bytesRead);
                    }
                } while (bytesRead > 0);
            }
            catch (Exception e)
            {
                throw e;
            }
            finally
            {
                fileStream.Close();
            }
            return filePath;
        }

And I was getting the same error in the bytesRead = cgStream.Read(buffer, 0, CHUNK_SIZE); line. Do you still think it is a CopyToAsync problem? I doubt it because when I put a break point I can see that stream is not readable as Read property is false before even going to CopyToAsync method.

I tried CopyTo instead of CopyToAsync and it did not help. Screenshot: