0

I have one application provide RESTful service facing performance issue when TPS increase from 5 TPS to 10 TPS, the 95% response time drop from 500++ ms to 26s.

My Application is web application use java (7) + Spring (3.3x) + Apache HTTP Client (4.5.12) and deploy on tomcat 7. Client sends JSON request via web layer use Apache web server as reverse proxy route the request to tomcat 7 (ajp) on the same machine.

I am refer to https://howtodoinjava.com/spring-boot2/resttemplate/resttemplate-httpclient-java-config/ to use PoolingHttpClientConnectionManager to create RestTemplate for POST request for another back-end mock service.

It works fine with 5 TPS and can see the application able to reuse HTTP connection. but when I increase request traffic to 10 above TPS, then application has poor performance sometime even get 26++s response time.

Connection pool with http config as below:

  DefaultMaxPerRoute = 200
  CONNECT_TIMEOUT = 30000
  SOCKET_TIMEOUT = 60000
  MAX_TOTAL_CONNECTIONS = 300
  DEFAULT_KEEP_ALIVE_TIME_MILLIS=20 * 1000
  CLOSE_IDLE_CONNECTION_WAIT_TIME_SECS =30

There have one idleConnectionMonitor to check and close expire/idle connection every 20s. I am not sure why it has so poor performance than I expect. It looks thread lock and waiting something and I print pool info via getTotalStats() and the HTTP connection is not exceed max total. Tomcat 7 is use default setting. e.g. thread pool is 200 as max Do not have experience on Apache reverse proxy setting. Don't know the issue is on Apache web (reverse proxy) or Tomcat7 setting or my application layer. Please advise on it and thanks a lot.

BTW, use gatling 3x as load test client.

Baodi Di
  • 540
  • 2
  • 6
  • 16
  • I used the RestTemplate with httpClient but not DI and Singleton object of restTemplate, so application will create fresh new http connection per request. I compare the two cases with same load and found. The apache2 has different behaviour: if use original approach, number of connection from apache2 to tomcat is change dynamic such ESTABLISH, TIME_WAIT if use netstat -na to monitor . but if use new approach, the apache2 keep the CONNECTION as establish between apache2 and TOMCAT port 8009. even application pool manage close for those idle/expired connection with downstream mock service. – Baodi Di Apr 10 '20 at 04:11
  • Check if [this answer](https://stackoverflow.com/questions/50890024/correct-way-to-implement-http-connection-pooling/50901789#50901789) helps a bit. – LMC Apr 10 '20 at 18:14

0 Answers0