26

I'm trying to fetch a series of files via HTTP, using HttpWebRequest. The first request goes through fine, but the second time through the same code GetResponse() hangs and times out. WireShark shows that no HTTP traffic is being sent for the second request, so it would appear that it's an API problem.

After some investigation, I've discovered that it has to do with specifying the content-length: if I leave this out, then the code works fine.

My code is:

HttpWebRequest  httpWebRequest = ConfigureRequest();

using (WebResponse webResponse = httpWebRequest.GetResponse())
    // On the second iteration we never get beyond this line
{
    HttpWebResponse httpWebResponse = webResponse as HttpWebResponse;

    using (Stream webResponseStream = httpWebResponse.GetResponseStream())
    {
        if (webResponseStream != null)
        {
            // Read the stream
        }
    }

    statusCode = httpWebResponse.StatusCode;
    httpWebResponse.Close();
}

The symptoms seem very similar to this question and this question, but in both cases the advice given is to dispose of the WebResponse, which I'm already doing.

Edit In response to Gregory, here is ConfigureRequest():

private HttpWebRequest ConfigureRequest()
{
    string          sUrl            = CreateURL(bucket, key);
    HttpWebRequest  httpWebRequest  = WebRequest.Create(sUrl) as HttpWebRequest;

    httpWebRequest.AllowWriteStreamBuffering = false;
    httpWebRequest.AllowAutoRedirect = true;
    httpWebRequest.UserAgent = this.m_sUserAgent;
    httpWebRequest.Method = "GET";
    httpWebRequest.Timeout = this.m_iTimeout;

    // *** NB: This line was left out of my original posting, and turned out to be
    // crucial
    if (m_contentLength > 0)
        httpWebRequest.ContentLength = m_contentLength;

    httpWebRequest.Headers.Add(StaticValues.Amazon_AlternativeDateHeader, timestamp);
    httpWebRequest.Headers.Add(StaticValues.HttpRequestHeader_Authorization, StaticValues.Amazon_AWS + " " + aWSAccessKeyId + ":" + signature);

    return httpWebRequest;
}

Edit: It turns out I committed the cardinal sin of removing code from my question that I hadn't verified was unrelated to the problem. I had removed the following lines:

    if (m_contentLength > 0)
        httpWebRequest.ContentLength = m_contentLength;

because I thought that content-length would never be specified for a GET request. It turns out I was wrong. Removing this line fixes the problem.

The only question I now have is why? I think the content length that is specified is correct, though it's possible it's off by one. Would specifying a content length that is too short prevent the full download from taking place and cause the connection to be left open? I would have expected that Close() and / or Dispose() ought to kill the connection anyway.

Community
  • 1
  • 1
Tim Martin
  • 3,618
  • 6
  • 32
  • 43
  • Can you post ConfigureRequest()? – Gregory Jan 14 '10 at 13:03
  • Also, re: http://stackoverflow.com/questions/1386628/webrequest-getresponse-locks-up have you tried configuring the value JSkeet mentions to something higher, ie 4 or 8, and see if this changes anything? – Gregory Jan 14 '10 at 13:05
  • @Tim Martin: What line did u remove? The if-clause and the setting of the ContentLength? So you do not set the content length at all? I tried removing the setting of ContentLenght, but I still have the same problem. – Ted Dec 04 '11 at 19:11
  • 1
    I had similar problem. This solved my problems: http://stackoverflow.com/questions/15181050/httpwebrequest-timing-out-on-third-try-only-two-connections-allowed-http-1-1 – Jaanus Mar 03 '13 at 09:41

5 Answers5

13
httpWebRequest.Abort(); // before you leave

Will solve!

Take a look at the resource.

  • Any chance of more information on _why_ this works? I have been pulling my hair out trying to find undisposed `WebHttpResponse` objects with no luck. Your solution has fixxed the problem - but why? – David Poxon May 24 '19 at 05:18
  • @DavidPoxon man, I had this same problem and calling this method was the answer, but I don`t remember the situation... take a look at this [resource](https://learn.microsoft.com/pt-br/dotnet/api/system.net.httpwebrequest.abort?view=netframework-4.8#System_Net_HttpWebRequest_Abort) –  May 24 '19 at 17:34
  • @DavidPoxon They use a `TimeoutCallback` to call `Abort` when the request timeout. –  May 24 '19 at 17:37
8

Make sure you are creating a NEW HttpWebRequest each time. Quote from the MSDN reference on GetResponse method :

Multiple calls to GetResponse return the same response object; the request is not reissued.

Update 1: Ok. How about if you try closing the webResponseStream too - you currently don't do that, you're only closing the webResponse (just trying to rule things out). I'd also dispose of the httpWebResponse too (not just the WebResponse)

Update 2: The only other thing I can suggest is to have a look at the following articles that I looked at when I did something similar to what you're doing:
http://arnosoftwaredev.blogspot.com/2006/09/net-20-httpwebrequestkeepalive-and.html
http://www.devnewsgroups.net/dotnetframework/t10805-exception-with-httpwebrequest-getresponse.aspx

The only noticeable things I do different are:
- set webrequest.KeepAlive = false
- I don't have the connection management stuff in the config (I don't loop round to make a series of requests)

AdaTheDev
  • 142,592
  • 28
  • 206
  • 200
  • Each time round the loop I'm calling WebRequest.Create(url) to create a new request. – Tim Martin Jan 14 '10 at 12:27
  • Thanks for the additional suggestions. I'm now closing the stream (it was already being Dispose()ed), and I've also wrapped the HttpWebResponse in a using block as well. – Tim Martin Jan 14 '10 at 13:19
  • I have tried the KeepAlive=false and the "using"-approach, I still get the same problem. It looks like GetResponse is the culprit, but the problem is that the POST (the second request) isnt sent at all (as MSDN states), thus the GetResponse is waiting for an answer. – Ted Dec 05 '11 at 10:16
1

Documentation for HttpWebRequest.ContentLength says:

Any value other than -1 in the ContentLength property indicates that the request uploads data and that only methods that upload data are allowed to be set in the Method property.

After the ContentLength property is set to a value, that number of bytes must be written to the request stream that is returned by calling the GetRequestStream method or both the BeginGetRequestStream and the EndGetRequestStream methods.

I suspect it hangs because you're setting the content length but then not uploading any data. One would think that it should throw an exception in that case, but apparently it doesn't.

Community
  • 1
  • 1
Jim Mischel
  • 131,090
  • 20
  • 188
  • 351
0

Plase try use:

         HttpRequestCachePolicy noCachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.NoCacheNoStore);
         httpWebRequest.CachePolicy = noCachePolicy;
kris164
  • 11
  • 2
-1

'You need to initialize the webrequest and webresponse objects with a valid URI:

        Dim request As WebRequest = WebRequest.Create("http://google.com")
        Dim response As WebResponse = request.GetResponse()

        Function UrlExists(ByVal URL As String) As Boolean
            Try
                request = WebRequest.Create(URL)
                response = request.GetResponse()
            Catch ex As Exception
                Return False
            Finally
                response.Close()
            End Try
            Return True
        End Function
Aarón Altamirano
  • 11
  • 1
  • Can you explain how this answers the question? – Nathan Tuggy Mar 05 '15 at 00:36
  • Yes, sorry! I had the same problem. You need to initialize the webrequest and webresponse objects with a valid URI: Dim request As WebRequest = WebRequest.Create("http://google.com") Dim response As WebResponse = request.GetResponse() Finaly you need to close the response object: Finally response.Close() I hope it helps you :D – Aarón Altamirano Mar 05 '15 at 00:42