How do I prevent Socket/Port Exhaustion?

Question

I am attempting to performance test a website by hitting it with requests across multiple threads. Each thread executes n times. (in a for loop)

However, I am running into problems. Specifically the WebException ("Unable to connect to remote server") with the inner exception:

An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full 127.0.0.1:52395

I am attempting to run 100 threads at 500 iterations per thread.

Initially I was using HttpWebRequest in System.Net to make the GET request to the server. Currently I am using WebClient as I assumed that each iteration was using a new socket (so 100 * 500 sockets in a short period of time). I assumed WebClient (which is instantiated once per thread) would only use one socket.

I don't need 50 000 sockets open at once, as I would like to send the GET request, receive the response, and close the socket, freeing it for use in the next loop iteration. I understand that it would be a problem to

However, even with WebClient, a bunch of sockets are being requested resulting in a bunch of sockets in TIME_WAIT mode (checked using netstat). This causes other applications (like internet browsers) to hang and stop functioning.

I can operate my test with less iterations and/or less threads, as it appears the sockets do eventually exit this TIME_WAIT state. However, this is not a solution as it doesn't adequately test the abilities of the web server.

Question:

How do I explicitly close a socket (from the client side) after each thread iteration in order to prevent TIME_WAIT states and socket exhaustion?

Code:

Class that wraps the HttpRequest

Edit: Wrapped WebClient in a using, so a new one is instantiated,used and disposed for every iteration. The problem still persists.

  public sealed class HttpGetTest : ITest {
    private readonly string m_url;

    public HttpGetTest( string url ) {          
        m_url = url;
    }

    void ITest.Execute() {
        using (WebClient webClient = new WebClient()){
            using( Stream stream = webClient.OpenRead( m_url ) ) {          
            }
        }
    }
}

The part of my ThreadWrapperClass that creates a new thread:

public void Execute() {
    Action Hammer = () => {
        for( int i = 1; i <= m_iterations; i++ ) {
            //Where m_test is an ITest injected through constructor
            m_test.Execute();
        }       
    };
    ThreadStart work = delegate {
        Hammer();
    };
    Thread thread = new Thread( work );
    thread.Start();
}

One thing to consider is not to test using the "drink from the fire hose" method. You should start slowly and ramp up the requests/sec to a fixed maximum to properly test your system. Then you can increase your maximum over a number of runs until you find the limits. Unlimited web requests will tell you very little. — Gray, Jul 19 '12 at 21:20
Keep in mind there are only ~65000 available ports, and not all of them can be used for outgoing connections. So you would need to use multiple IPs/NICs to do the 50000 connections that you are trying to do — Stefan H, Jul 19 '12 at 21:20
@StefanH I understand this would be a concern if I was executing a large number of threads, however, once an iteration of the loop is done I don't need the socket anymore, but it sticks around, causing the next iteration to open a new one. I'm looking to find a way to prevent that — James, Jul 19 '12 at 21:23
You are doing a using on the stream you are getting back, but not your web client. You could try a using statement on your WebClient as well so that it gets disposed. Or manually dispose it once you are done reading. — Stefan H, Jul 19 '12 at 21:27

spender · Accepted Answer · 2012-07-19T22:12:43.530

Do you understand the purpose of TIME_WAIT? It's a period during which it would be unsafe to reuse the port because lost packets (that have been successfully retransmitted) from the previous transaction might yet be delivered within that time period.

You could probably tweak it down in the registry somewhere, but I question if this is a sensible next step.

My experience of creating realistic load in a test environment have proved very frustrating. Certainly running your load-tester from localhost is by no means realistic, and most network tests I have made using the .net http apis seem to require more grunt in the client than the server itself.

As such, it's better to move to a second machine for generating load on your server... however domestic routing equipment is rarely up to the job of supporting anywhere near the number of connections that would cause any sort of load on a well written server app, so now you need to upgrade your routing/switching equipment as well!

Lastly, I've had some really strange and unexpected performance issues around the .net Http client API. At the end of the day, they all use HttpWebRequest to do the heavy lifting. IMO it's nowhere near as performant as it could be. DNS is sychronous, even when calling the APIs asynchronously (although if you're only requesting from a single host, this isn't an issue), and after sustained usage CPU usage creeps up until the client becomes CPU constrained rather than IO constrained. If you're looking to generate sustained and heavy load, any request-heavy app reliant on HttpWebRequest is IMO a bogus investment.

All in all, a pretty tricky job, and ultimately, something that can only be proved in the wild, unless you've got plently of cash to spend on an armada of better equipment.

[Hint: I got much better perfomance from my own client written using async Socket apis and a 3rd party DNS client library]

I appreciate this post. I'll try the separate hardware/office resource allocation path, but I'd also like to try to implement a tech solution that doesnt require additional resources. What library did you use? Are other technologies (I'm thinking C++) "better"? — James, Jul 19 '12 at 22:02
Well, DNS isn't going to be an issue for you, but you might find this interesting: http://stackoverflow.com/questions/11480742/dns-begingethost-methods-blocking . I've found you can get much better performance from chatting HTTP using the .net Socket api. I wrote a small library myself (for spidering), but can't share because it belongs to my company, and it only implements a very limited subset of HTTP. It does however make .net very fast indeed when doing HTTP, hitting IO limits while virtually idling. — spender, Jul 19 '12 at 22:06

score 3 · Answer 2 · answered Jul 19 '12 at 22:15

Q: How do I explicitly close a socket ... in order to prevent TIME_WAIT states?

A: Dude, TIME_WAIT is an integral - and important! - part of TCP/IP itself!

You can tune the OS to reduce TIME_WAIT (which can have negative repercussions).

And you can tune the OS to increase #/ephemeral ports:

http://msdn.microsoft.com/en-us/library/aa560610%28v=bts.20%29.aspx

Here's a link on why TIME_WAIT exists ... and why it's a Good Thing:

http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html

Martin James · Answer 3 · 2012-07-19T21:46:37.650

It's not an issue of closing sockets or releasing resources in your app. The TIME _WAIT is a TCP stack timeot on released sockets to prevent their re-use until such time as it is virtually impossible for any packets 'left over' from a previous connection to that socket to not have expired.

For test purposes, you can reduce the wait time from the default, (some minutes, AFAIK), to a smaller value. When load-testing servers, I set it at six seconds.

It's in the registry somewhere - you'll find it if you Google.

Found it:

Change TIME_WAIT delay

Stefan H · Answer 4 · 2012-07-19T21:35:33.487

1

It looks like you are not forcing your WebClient to get rid of the resources that it has allocated. You are performing a Using on the stream that is returned, but your WebClient still has resources.

Either wrap your WebClient instantiation in a using block, or manually call dispose on it once you are done reading from the URL.

Try this:

public sealed class HttpGetTest : ITest {
    private readonly string m_url;

    public HttpGetTest( string url ) {
        m_url = url;        
    }

    public void ITest.Execute() {
        using( var m_webClient = new WebClient())
        {
            using( Stream stream = m_webClient.OpenRead( m_url ) ) 
            {

            }
        }
    }
}

edited Jul 19 '12 at 21:35

answered Jul 19 '12 at 21:29

Stefan H

6,635
4
24
35

I initially left it "unwrapped" as I assumed it would only create 1 socket in it's lifetime. I edited my code, (as reflected in my above edit) but the problem still persists. – James Jul 19 '12 at 21:34
@James I'm not certain what it is then, sorry I couldn't have been of more help. – Stefan H Jul 19 '12 at 21:37

score 0 · Answer 5 · answered Dec 27 '16 at 16:40

0

You don't need to mess around with TIME_WAIT to accomplish what you want.

The problem is that you are disposing the WebClient every time you call Execute(). When you do that, you close the socket connection with the server and the TCP port keeps busy for the TIME_WAIT period.

A better approach is to create the WebClient in the constructor of your HttpGetTest class and reuse the same object throughout the test.

WebClient uses keep alive by default and will reuse the same connection for all its requests so in your case there will be only 100 opened connections for this.

answered Dec 27 '16 at 16:40

andrecarlucci

5,927
7
52
58

1. This is what the original version of the question's code did (and the question still talks about "I assumed WebClient (which is instantiated once per thread) would only use one socket." 2. Keep-alive only reuses the connection if you're making a second connection to the same hostname. Question is about reusing the client socket after closing the connection. – Ben Voigt Dec 27 '16 at 16:46
Hmm... the question was edited, I didn't see the original code. 1. Yes, this is correct as I said. 2. Yes, same host, how else could it be? Also, he didn't say there were many hosts and even if it was a webfarm, 100 threads would be probably enough to hit all the the servers (assuming it's far less than 100). Probably there is something else going on, WebClient should behave correctly if you reuse it. And the question is "How do I prevent Socket/Port Exhaustion?": using WebClient (or maybe HttpClient) correctly will prevent port exhaustion. – andrecarlucci Dec 27 '16 at 18:16

How do I prevent Socket/Port Exhaustion?

5 Answers5

Linked

Related