How to detect HTTPRequest Failure

Question

I am sending 100000 Requests and in order to check if all the requests has been sent successfully , I have developed a simple web page that counts the number of requests that has been sent.

The problem is this that the receiver counts less than 50000 and I also can not detect which one of them has been failed in order to send them again as the sender gets statusscode=OK for all of them and also no exception has detected.

I also tried it after removing webReq.Method="HEAD" but had no effect.

Any hints is appreciated.

Here is the sender's code:

try
{
    var content = new MemoryStream();
    var webReq = (HttpWebRequest)WebRequest.Create(url);
    webReq.Method = "HEAD";
    using (WebResponse response = await webReq.GetResponseAsync())
    {
        HttpWebResponse res = (HttpWebResponse)response;
        if (res.StatusCode != HttpStatusCode.OK)
        {
            UnsuccessfulURLsPhase1.Add(url);
        }
    }
}
catch (Exception e)
{
    UnsuccessfulURLsPhase1.Add(url);
}

This is receiver's code:

    protected void Page_Load(object sender, EventArgs e)
    {
        try
        {
            if (!IsPostBack)
            {
                counter1++;
                txtCounter.Text = counter1.ToString();
            }
        }
        catch (Exception ex)
        {
            Debug.WriteLine("\nException raised!");
            Debug.WriteLine("Source :{0} ", ex.Source);
            Debug.WriteLine("Message :{0} ", ex.Message);
        }
    }

score 0 · Answer 1 · edited May 23 '17 at 12:22

Your Page_Load is swallowing the exceptions so the server is always returning 200, OK.

You have to let the exception be thrown or you have to explicitly set the response status when an error occurs. Found in here How to send a Status Code 500 in ASP.Net and still write to the response? that TrySkipIisCustomErrors should be set.

Something like

protected void Page_Load(object sender, EventArgs e)
{
    try
    {
        if (!IsPostBack)
        {
            counter1++;
            txtCounter.Text = counter1.ToString();
        }
    }
    catch (Exception ex)
    {
        Debug.WriteLine("\nException raised!");
        Debug.WriteLine("Source :{0} ", ex.Source);
        Debug.WriteLine("Message :{0} ", ex.Message);

       // Raise the exception
       // throw;   
       // or assign the correct status and status code
       Response.Clear();     
       Response.TrySkipIisCustomErrors = true
       Response.ContentType = "text/plain";
       Response.StatusCode = (int)HttpStatusCode.InternalServerError;
       Response.Write(ex.Message);

       // Send the output to the client.
       Response.Flush();
    }
}

(Hope it helps, its been a long time since I have done something in WebForms :S )

Thanks for your comment, I added your code to the exception but unfortunately it doesn't hit, when it is missing few of the calls , the breakpoint at the exception shows nothing. — albert sh, Sep 29 '15 at 17:56

score 0 · Answer 2 · answered Sep 29 '15 at 15:16

You're sending these out as quickly as possible? IANA suggests (and recent editions of Windows respect) using the range 49152 to 65535 for ephemeral ports, which are ports that are reserved to receive the reply from IP socket connections. This means that there are 16383 ports available, each of which must be left in a TIME_WAIT state for (IIRC) 120 seconds after the connection is closed.

In perfect conditions (and with routing equipment that can sustain thousands of simultaneous connections... a cheap SOHO router will probably overheat and become unreliable as it runs out of memory), you're going to be limited to a maximum of around 16000 requests every two minutes.

In practice, HttpWebRequest (and therefore WebClient) will maintain only a specific number of connections to a specific host, and pipeline requests over those connections, so without tweaking ServicePointManager, or the ServicePoint associated with the host you're trying to hit, you're going to have an awful amount of queueing to squeeze 100000 requests through these connections. It's likely that you'll hit timeouts somewhere down this path.

Thank you for the comment. I am missing around 4000 from 100K, but as you said it can only get 16000 within two minutes, how it happens?What is the solution anyway? — albert sh, Oct 05 '15 at 17:23

How to detect HTTPRequest Failure

2 Answers2