We have two of these servers, were we are observing these errors. Both talk to the master RDS db. The TPS value has recently risen and touching almost 50 TPS. CPU usage on application servers are around 40%.
I am getting lots of these errors -
2019-09-02 00:00:12,714 65086940 [XNIO-3 task-18] INFO c.c.p.c.s.CustomRemoteTokenService [CustomRemoteTokenService.java:57] - API : service/api/path/profile/summary property api : system;;
2019-09-02 00:00:12,763 65086989 [XNIO-3 task-18] WARN o.h.e.jdbc.spi.SqlExceptionHelper [SqlExceptionHelper.java:127] - SQL Error: 0, SQLState: 08S01
2019-09-02 00:00:12,764 65086990 [XNIO-3 task-18] ERROR o.h.e.jdbc.spi.SqlExceptionHelper [SqlExceptionHelper.java:129] - Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
2019-09-02 00:00:12,765 65086991 [XNIO-3 task-18] ERROR io.undertow.request [LoggingExceptionHandler.java:80] - UT005023: Exception handling request to /service/path/to/profile/summary
java.lang.NoClassDefFoundError: org/hibernate/internal/util/JdbcExceptionHelper
Almost, 3k in 5 mins, in one server, while the other server has 0 today. Interestingly, the errors are typically high on one of the application servers, while both are expected to handle similar load.
Db configuration values are -
show global variables like "%timeout%"
-> ;
+-----------------------------+----------+
| Variable_name | Value |
+-----------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| have_statement_timeout | YES |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 50 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| rpl_stop_slave_timeout | 31536000 |
| slave_net_timeout | 60 |
| wait_timeout | 28800 |
+-----------------------------+----------+
I checked this question Database Connection to MySQL times out even after setting c3p0.testConnectionOnCheckout=true as well, however, since I don't have all the hibernate configs defined in my case, I am not sure if I need to tweak those.
I have only the following defined in my persistence.xml file -
<property name="hibernate.show_sql" value="true"/>
<property name="hibernate.format_sql" value="true"/>
<property name="hibernate.ejb.naming_strategy" value="org.hibernate.cfg.ImprovedNamingStrategy"/>
<property name="hibernate.connection.charSet" value="UTF-8"/>
