Background
I'm running a laravel 5.3 powered web app on nginx. My prod is working fine (running on an AWS t2.medium) and my staging has been running fine until it recently got overloaded. Staging is a t2.micro.
Problem
The problem happened when started trying to hit the api endpoints, and started getting this error:
503 (Service Unavailable: Back-end server is at capacity)
using htop we got the following:
so we found that our beanstalk queues are taking insane amounts of memory.
What I have tried
we used telnet to peek into what's going inside beanstalk:
$~/beanstalk-console$ telnet localhost 11300
Trying 127.0.0.1...
Connected to staging-api-3.
Escape character is '^]'.
stats
OK 940
---
current-jobs-urgent: 0
current-jobs-ready: 0
current-jobs-reserved: 0
current-jobs-delayed: 2
current-jobs-buried: 0
cmd-put: 451
cmd-peek: 0
cmd-peek-ready: 0
cmd-peek-delayed: 0
cmd-peek-buried: 0
cmd-reserve: 0
cmd-reserve-with-timeout: 769174
cmd-delete: 449
cmd-release: 6
cmd-use: 321
cmd-watch: 579067
cmd-ignore: 579067
cmd-bury: 0
cmd-kick: 0
cmd-touch: 0
cmd-stats: 1
cmd-stats-job: 464
cmd-stats-tube: 0
cmd-list-tubes: 0
cmd-list-tube-used: 0
cmd-list-tubes-watched: 0
cmd-pause-tube: 0
job-timeouts: 0
total-jobs: 451
max-job-size: 65535
current-tubes: 2
current-connections: 1
current-producers: 0
current-workers: 0
current-waiting: 0
total-connections: 769377
pid: 1107
version: 1.10
rusage-utime: 97.572000
rusage-stime: 274.560000
uptime: 1609870
binlog-oldest-index: 0
binlog-current-index: 0
binlog-records-migrated: 0
binlog-records-written: 0
binlog-max-size: 10485760
id: 906b3629b01390dc
hostname: staging-api-3
nothing there seems to be concerning..
Question
I would like to have a more transparent look into whats going on in these jobs (ie what are the jobs exactly?) I know Laravel Horizon provides such services but only comes on Laravel 5.5. I researched what queue monitors are out there and tried to install beanstalk console. Right now when I installed it.. I'm getting 52.16.%ip%.%ip% took too long to respond. which I think is expected considering that the whole machine is already jammed.
I figure if I reboot the machine I can install beanstalk_console just fine, but then i'll lose the opportunity to investigate what's causing the problem this time around, sine it's a rare occurrence.. What else can I do to investigate and see what exactly are the jobs that are draining the CPU and why
Update
I restarted the instance, and the apis work now, but i'm still getting CPU is at 100%.. what am I missing?
