0

Our API which is having 600 request per min access, the API setup is using two dedicated servers one for php process and second for mariadb. We are using 14 core, 128 ram servers for both.

The communication between api and database server is happening via local network, when ever there is large access towards api we have noticed two errors one on api larvel log and second on mysql server side.

production.ERROR: Symfony\Component\Debug\Exception\FatalErrorException: Maximum execution time of 60 seconds exceeded

[Warning] Aborted connection 48520867 to db: 'database-name' user: 'root' host: '172.16.1.2' (Got timeout reading communication packets)

We have managed to fix the api error by making two changes, php max_execution_time to 120 seconds and setting kernel parameter net.ipv4.tcp_tw_recycle = 1,

Once this is done we are now facing connection drop from our Office ISP with dedicated IP address, we have more than 100 users connecting from same IP.

Once the kernel parameter "net.ipv4.tcp_tw_recycle = 1" is disabled our office IP can access it cool. We already have "net.ipv4.tcp_tw_reuse = 1".

What is the possible solution for these ? I could also see a stackoverflow discussion which is almost similar to the same issue. But still unable to find a solution for the same.

Dropping of connections with tcp_tw_recycle

nisamudeen97
  • 519
  • 4
  • 22

2 Answers2

0

I think the problem arise due to the more number of connections between the PHP and Maria DB.

Please increase the number of request that can be handled by Apache, and increase the buffer memory size of the Maria DB.

In Mysql we can increase the below mentioned parameters (Not worked with Maria DB till now)

/etc/my.cnf
***innodb_buffer_pool_size = 384M***
key_buffer = 256M
query_cache_size = 1M
query_cache_limit = 128M
thread_cache_size = 8
***max_connections = 400***
innodb_lock_wait_timeout = 100

Please have a try.

Ajai Soman
  • 1
  • 1
  • We have already done those, innodb_buffer_pool_size = 90GB innodb_thread_concurrency = 48 imax_connections = 65535 connect_timeout = 5 wait_timeout = 600 max_allowed_packet = 1024M thread_cache_size = 128 sort_buffer_size = 4M bulk_insert_buffer_size = 16M tmp_table_size = 32M max_heap_table_size = 32M – nisamudeen97 Jun 30 '17 at 10:53
  • Have you increased the Apache web server MaxClients to process value? – Ajai Soman Jun 30 '17 at 13:35
  • http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients https://stackoverflow.com/questions/3389496/how-do-you-increase-the-max-number-of-concurrent-connections-in-apache – Ajai Soman Jun 30 '17 at 13:36
0

The setting "net.ipv4.tcp_tw_recycle = 0" has completely fixed the dropping of connections. Working good. We can set net.ipv4.tcp_tw_recycle to 0 by using the below command.

sysctl -w net.ipv4.tcp_tw_recycle=0

Reference discussions :-

tcp_tw_reuse vs tcp_tw_recycle : Which to use (or both)?

https://serverfault.com/questions/234534/is-it-dangerous-to-change-the-value-of-proc-sys-net-ipv4-tcp-tw-reuse

nisamudeen97
  • 519
  • 4
  • 22