2

I'm suddenly having intermittent 2003 connection errors and I can't find anything that has changed that can explain it. These servers have been running smoothly for years and suddenly two days ago these errors started showing up in my logs. The application successfully connects to and queries the database almost all the time but sporadically hangs when trying to connect after several queries in quick succession. The connection timeouts always happen at random intervals. It worked just fine before two days ago and the code hasn't changed in 4 years.

Things I've tried that didn't work:

  • Checked all InnoDB tables for corruption
  • Increased the back_log value in my.cnf from 50 to 1000 (and restarted MySQL)
  • Increased the max_connections value in my.cnf from 100 to 200 (and restarted MySQL)
  • Increased net.core.somaxconn from 128 to 1024
  • Increased net.ipv4.tcp_max_syn_backlog from 1024 to 8192
  • Verified there are no firewall rules blocking or throttling connections
  • Flushed the MySQL query cache
  • Cleared the MySQL query cache
  • Restarted the application server and database server (there are two separate servers)
  • Asked it nicely to simply start working again

My servers are EC2 instances in AWS and nothing about their configuration has been changed. There's no unusual traffic and the CPU, memory, and disk usage are well below 80%. Please help!

Olaf Kock
  • 46,930
  • 8
  • 59
  • 90
aecend
  • 2,432
  • 14
  • 16
  • This is not a programming problem. Ask on [dba.se] – Tangentially Perpendicular Sep 14 '21 at 23:35
  • Additional information request. AWS Model/Type, any SSD or NVME devices on MySQL Host server? Post on pastebin.com and share the links. From your SSH login root, Text results of: B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME, if you can get there. C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; E) STATUS; not SHOW STATUS, just STATUS; G) SHOW ENGINE INNODB STATUS; H) SELECT name, count FROM information_schema.innodb_metrics ORDER BY name; for server workload tuning analysis to provide suggestions. – Wilson Hauck Sep 14 '21 at 23:40
  • @aecend Your profile has an invalid URL listed. – Wilson Hauck Sep 14 '21 at 23:47
  • Did you say please? – David Gerstenmier Sep 15 '21 at 15:32
  • @DavidGerstenmier Do you mean like you did on this https://stackoverflow.com/questions/53684461/ ? – Wilson Hauck Sep 18 '21 at 18:36

0 Answers0