1

I have an Elastic Beanstalk instance which then runs out of memory starts giving 500 errors and the health goes to degraded which is expected behavior. Now when the memory is released server health is back to normal the HTTPS requests are stalling. There is no CPU or memory activity on the server but server still fails to load 2 out of 5 requests. The issue is resolved once i restart the Elastic Beanstalk instance.

I checked the error logs but there are no records seems as if the requests are not hitting the server they are being blocked at the load balancer. Is any one facing this issue? I tried running few different applications on the Elastic Beanstalk instance but results are the same.

Any suggestions or pointing me to docs will be appreciated as i didn't find anything in their docs about this.

Thanks

stefansundin
  • 2,826
  • 1
  • 20
  • 28
Edi 0
  • 212
  • 5
  • 22
  • Have you logged in into the instance, and tried to check which process takes up all the memory? – Marcin Jun 16 '20 at 00:11
  • Yes i have installed Htop on the server and there is a database operation that causes the out of memory error but when this is fixed then there is hardly any activity on the server but some requests are still failing which makes me curious. – Edi 0 Jun 16 '20 at 00:17

1 Answers1

3

It is possible that some important process was OOM killed, and has not started back up properly. You can look at this question for help on that: Finding which process was killed by Linux OOM killer

It may be possible to simply reboot the server, and everything may start back up properly again.

The best solution is to not run out of memory in the first place. If you are unable to upgrade the server, perhaps due to cost reasons, then consider adding a swap file.

I have an Elastic Beanstalk application in which I add a swap file using an ebextension.

# .ebextensions/01-swap.config

commands:
  "01-swap":
    command: |
      dd if=/dev/zero of=/var/swapfile bs=1M count=512
      chmod 600 /var/swapfile
      mkswap /var/swapfile
      swapon /var/swapfile
      echo "/var/swapfile none swap sw 0 0" >> /etc/fstab
    test: test ! -f /var/swapfile

https://github.com/stefansundin/rssbox/blob/1e40fe60f888ad0143e5c4fb83c1471986032963/.ebextensions/01-swap.config

stefansundin
  • 2,826
  • 1
  • 20
  • 28