2

I have a webapp with a Rails API running in a Puma server with a MySQL database. I´m running a batch process that is writing a lot of logs to a log file. In a specific moment after some time running the Puma server goes down. In the logs everythink looks fine until aparently with no reason it gets down. But I don´t see any error. Neither in the puma.err. Not sure if it´s related to the server or maybe the database (something related with the pool?). I´m blind. I don´t know where to look at to debug the problem.

UPDATE: I think I have narrowed down the problem. I have realised that the Puma server gets down always when trying to load the same element. I tried to make a test in development and it works there. So, I think I know what is killing it. I´m not sure but strongly suspect the element I´m trying to load in my batch process run a lot of queries within a single transaction in the MySQL database. I think somehow the transaction is not able to process all the queries and for some reason the Puma server gets down. I don´t know if this makes sense, but this is my main suspect. I have read something about transaction sizes and log files How do I determine maximum transaction size in MySQL?

My new question is: if this is really happening, can I see an error related to this in any log file (Puma or Mysql)?

UPDATE 2: I attach enviroment information:

DEVELOPMENT: MacOS, Processor: 2,4GHz, RAM: 8GB

PRODUCTION: Ubuntu14, Processor: 2,5GHz, RAM: 3,7GB (AWS instance) Regarding the Puma configuration, I´m not very skilled but I´m starting the server with a config file with just two lines in both development and configuration environments:

stdout_redirect 'path_to_log_file.log', 'path_to_error_file.log', true
bind 'unix:///tmp/puma.sock'

When starting up I neither see and workers configuration, so I assume I have only one worker and the default number of threads 0 to 16.

Rober
  • 5,868
  • 17
  • 58
  • 110
  • You need to find a way to reproduce the problem first. You say the logs looks fine, but can't you see some pattern from the last actions before it goes down? Configure the logger to be :debug instead of :info so you'll have even more info on the logs. You need to log everything you can. You can also add some kind of system monitor like monit or god to keep track of the server's process resurces (maybe the batch process takes too much memory and gets killed, who knows). You really need to find a way to reproduce it, if you just try things you'll never know if it's fixed or just a coincidence. – arieljuod Dec 13 '18 at 14:05
  • Are you sure this is server related? Try running the app using the iodine server (use the same concurrency settings you use with Puma) and see if the issue persists. This can rule out any server side code. – Myst Dec 14 '18 at 01:41
  • I´m not sure about anything. I just know the server gets down. I know it´s always in the same place, I mean in the logs. The point is that the logs don´t show anything, until it gets down everything looks good. I think it´s the next thing that it tries to process what kill it. But, not sure if it´s related to the server or the database. – Rober Dec 14 '18 at 11:49
  • @Myst please see my update. Do you think does it make sense? – Rober Dec 17 '18 at 19:49
  • @Rober , I think there are two big hints as to the root cause: 1. It only happens in production; 2. The logs don’t show any shutdown or error messages. I suspect it’s related to resource limitations on the production machine. Could you provide details about the production environment? Also, do you set up the development environment differently (i.e., use a single server worker instead of a cluster, etc’)? – Myst Dec 17 '18 at 19:59
  • @Myst please see my new update 2 with the info you requested. – Rober Dec 18 '18 at 07:09
  • @Rober , when I was suggesting to add the environment information to the question, I was actually referring to the configuration files (in addition to the general setup). Any logs in the AWS instance? Does the instance suddenly shutdown? – Myst Dec 20 '18 at 06:15
  • Which configuration files would you like to see. I haven´t seen any logs in AWS instance. But I think the instance doesn´t shutdown. I just need to restart the Puma server. – Rober Dec 20 '18 at 07:52

0 Answers0