0

The server run successfully at one time, but it hangs after days with no error logs. Then, all requests would not get the response.

This is the start command with options

sudo /opt/dev -Dhttps.port=443 -Dhttp.port=9000 -J-Xms3277m -J-Xmx3277m -J-XX:ParallelGCThreads=2 -J-Xmn2574M -J-XX:+UseConcMarkScMarkSweepGC -J-XX:+CMSClassUnloadingEnabled -J-server &

/opt/dev is the script file generated from activator stage

===========server info==========

linux: Ubuntu 14.04.5 LTS ram: 4G openjdk version "1.8.0_141"

===========process info========

PID   USER      PR   NI VIRT   RES     SHR   S   %CPU  %MEM  TIME     COMMAND

15037 root      20   0 5978800 2.280g  31216 S   0.0   58.3  63:33.82 java

===========port info ===================

tcp6  :::9000        :::*               LISTEN      15037/java

tcp6  :::443         :::*               LISTEN      15037/java

===========other info==========

 play version 2.3.2
 scala version 2.11.1

akka setting

 akka.jvm-exit-on-fatal-error = false
 play.akka.jvm-exit-on-fatal-error = false
 akka.default-dispatcher.fork-join-executor.pool-size-max =64
 akka.actor.debug.receive = on

===========================================

loui
  • 1

2 Answers2

1

These steps could help identify the problem.. or they could be just first steps in this direction.

  1. Try to start with adding -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/where/to/put/hprof according you start script params think you need to use -J-XX instead of -XX. This will create heap-dump in case of OOM.

  2. Add logging in endpoints (at start and at end) to be able to check if play receives request or even this does not happen.

  3. While you have unresponsive play, try to check open file descriptors and compare it with your limits. To check you can find pid of your java process and call sudo ls -al /proc/7333/fd/|wc -l to see your limits use ulimit -a.

  4. Would be nice to try to control akka queues. For the case if you use same dispatcher for frontend requests purposes and for some backoffice processing (dispatcher could be filled with long background tasks)

Evgeny
  • 1,760
  • 10
  • 11
1

I would do all the diagnostic steps that Evgeny suggested, plus:

  1. Change "akka.jvm-exit-on-fatal-error" and "play.akka.jvm-exit-on-fatal-error" to true, this may be masking your problem.

  2. Take a stack dump of the running process when it is in this state and use that to identify the problem or post it here. See How to get a complete stack trace of a running java program that is taking 100% cpu?

Rich
  • 15,048
  • 2
  • 66
  • 119