1

I've deployed a Java webap to AWS Elastic Beanstalk and every few days, sometimes more often I get notified:

AWS Elastic Beanstalk Notification - Adding instance i-5dc46494 to your environment

a few minutes later:

Message: Removed instance 'i-60a05f96' from your environment. (Reason: Instance is in 'shutting-down' state)

First, is there somewhere I can look to find out the reason it needs to replace the current instance?

Second, last time it did this I was logged in and did a 'dmesg' just before it shut down:

[4383952.557902] Out of memory: Kill process 1608 (java) score 796 or    sacrifice child
[4383952.557910] Killed process 1608 (java) total-vm:2496244kB, anon-rss:482052kB, file-rss:0kB

Probably the reason. What memory is it refering to here? the memory allocated to the JVM or the memory for the instance?

I also ran top and free -m

[ec2-user@ip-172-31-40-184 tomcat8]$ free -m
      total         used    free  shared  buffers   cached
Mem:  592           574     17      0     13        41
-/+ buffers/cache:  520     71
Swap:  0            0       0

here is pid 1608

[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[ 1608]    91  1608   624061   120513     332        0             0 java

Any insight on what is happening would be appreciated. thanks!

brad12s
  • 419
  • 2
  • 7
  • 17

1 Answers1

0

Are you using t1.micro instance? I saw your RAM size was less than 1 Gb. I didn't think it's enough to run Tomcat 8 with default setting.

But, you can do:

  • Upgrade your instance at least to t2.micro. It's cheaper and has better performance than t1.micro. (Do this first)
  • Limit your Tomcat 8 memory consume by setting JAVA_OPTS. Create a setenv.sh file using .ebextensions.

  • Add swap file/disk space. Read in this question. But someone warns in one of answers there:

    I have learned that it is counterproductive to use swap space on a EC2 micro instance because a) it is slow swap space and b) using it counts against IO which has also to be paid, allowing for an unexpected bill the next month.

Community
  • 1
  • 1
Edward Samuel
  • 3,846
  • 1
  • 22
  • 39
  • Yes i was using t1.micro, thanks for that info, I've up'd to t2 for now. Do you know if there is a way to determine why an instance was replaced or is that info lost when the instance is shut down? Any insight on the limit for tomcat8? – brad12s Aug 02 '15 at 04:06
  • It might be because your Tomcat is killed by [OOM Killer](http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html). So, no one will listen on port 80. By default in EB, Auto Scaling Group (ASG) will kill instances those failed to serve TCP port 80. – Edward Samuel Aug 02 '15 at 05:55