5

I have thousand proxies and thousand custom cookies.

Now I want to send requests with random proxy and using random cookie per request. I have tried to create new httpClientHandler and httpClient each time, but end up using all of my memory of my system.

After reading this article I understand I should reuse the same httpClient. But nor the properties of httpClient and httpClientHandler cannot be modified after the first request.

This instance has already started one or more requests. Properties can only be modified before sending the first request.

So I have used to change the webProxy.Address for changing proxy before sending requests. It works well but failed in the situation of multi-threading. It ends up using wrong proxy for the request. Any other ways to achieve my goal? THANKS.

Addtional Reference: HttpClientHandler / HttpClient Memory Leak

Quote from other users in the above reference:

Thanks for getting to the bottom of this. Unfortunately the HttpClient class does not meet my requirements then - due to the dynamic and unstable nature of public proxies, the objects HAVE to be often re-created. It appears HttpClient is just not a feasible solution for short-living connections -- changing the proxy settings requires re-constructing the HttpClientHandler, and thus the HttpClient. Either way, the objects should be able to live as long or short as needed without leaking; this definitely seems to be a flaw in the HttpClient.

shtse8
  • 1,092
  • 12
  • 20
  • Did you read all the negative comments at the link and the warning that using one Client is not thread safe? I would first try to figure out why your original solution had a memory leak. Then use your original approach and limit the number of parallel threads you run at one time so you do not use up all your computer resources. – jdweng Mar 07 '18 at 10:20
  • @jdweng [HttpClientHandler / HttpClient Memory Leak](https://stackoverflow.com/questions/27732546/httpclienthandler-httpclient-memory-leak?rq=1) I believe the memory leakage is due to the keep-alive connections. – shtse8 Mar 07 '18 at 10:25
  • Then close the Client when you are don with the client. A Keep-Alive packet is a packet with zero databaytes. So it just has an IP header and a TCP header. Which is very small. TCP memory leaks occur when retires occur. The retries are usually due to a PC running slow and not ack each TCP data packet quick enough. Usually the TCP retry time is set to 5 seconds with a max retry of 3. You can use PING to determine if retries are actually occurring. You can also see the reties using a sniffer like wireshark or fiddler. – jdweng Mar 07 '18 at 10:49

1 Answers1

9

For further investigation and reference for other comments, I believe this can solve my problem.

Instead of reusing the same handler and client, we should recreate it every request for thread safety.

In conclusion, to solve the memory leakage issue for short-lived requests. We can turn off the keep-alive so that the connection will be closed after the responses.

var client = new HttpClient();
client.DefaultRequestHeaders.ConnectionClose = true;

but It turns out this can add roughly extra 35ms (with long tails, i.e amplifying outliers) to each of your HTTP calls preventing you to take advantage of benefits of re-using a socket.

So we can change the keep-alive shorter to due with this instead of switching off keep-alive.

var sp = ServicePointManager.FindServicePoint(new Uri("http://foo.bar/baz/123?a=ab"));
sp.ConnectionLeaseTimeout = 60*1000; // 1 minute

Reference: Singleton HttpClient? Beware of this serious behaviour and how to fix it

shtse8
  • 1,092
  • 12
  • 20