I have Scala / Akka / Slick 3.x runnable that retrieve 25000 records in a Mysql 5.6 community DB. For each of them, it spawns an actor that will do 7 counts and return a case class containing those 7 values to the runner, which will, once finished save that as a csv file.
It works perfectly fine locally, it gets all the data, store them in memory, output them, close itself, everything is good. On my production DB, it's ok for about 40 seconds, I can see in the logs that it gets all the data until this happens :
[INFO] [03/24/2016 08:41:46.710] [indicator-runner-akka.actor.default-dispatcher-17] [akka://indicator-runner/user/user-usage-provider/$gO] Valid data set sent for xxxxxx
[INFO] [03/24/2016 08:41:46.711] [indicator-runner-akka.actor.default-dispatcher-17] [akka://indicator-runner/user/user-usage-provider/$fO] Valid data set sent for xxxxxx
[INFO] [03/24/2016 08:41:46.722] [indicator-runner-akka.actor.default-dispatcher-9] [akka://indicator-runner/user/user-usage-provider/$uN] Valid data set sent for xxxxxx
[INFO] [03/24/2016 08:41:46.731] [indicator-runner-akka.actor.default-dispatcher-12] [akka://indicator-runner/user/user-usage-provider/$hO] Valid data set sent for xxxxxx
[ERROR] [03/24/2016 08:41:46.823] [indicator-runner-akka.actor.default-dispatcher-6] [akka://indicator-runner/user/user-usage-provider/$kO] Should'nt receive this message : Failure(com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)$
At this point, i shutdown the actorsystem and return appropriate exit code, but this operation hasn't gone through.
I faced the maxQueue exception in slick which I fixed with :
executor = AsyncExecutor("HeavyLoad", numThreads = 24, queueSize = 10000)
I also tried :
echo 4096 > /proc/sys/net/core/somaxconn
And set max_connections to 500... No luck
Any ideas of what could go wrong ?
Thank you !
