48

I have a Maven project, which builds 6 separate Maven projects in Jenkins. The problem I face is that over the time the project build fails giving the 137 error code:

ERROR: Maven JVM terminated unexpectedly with exit code 137

The project could be built successfully using same Maven goals in the console, but in Jenkins it fails. By restarting Jenkins the problem can be resolved.

I have some static array lists. These lists are used for some test cases. Could this be a memory leak?

eis
  • 51,991
  • 13
  • 150
  • 199
user2822319
  • 541
  • 1
  • 5
  • 8
  • 3
    Did the build always fail in Jenkins, or only sometimes? Can you paste the error + error message? – Mateva Jul 28 '14 at 06:49
  • Recently it started to fail now all the builds are failing. – user2822319 Jul 28 '14 at 07:44
  • 1
    I've see the same thing and that is about all you get in terms of an error message. In my case, I'm running the maven-frontend-plugin and it's dying while trying to resolve npm dependencies. However, there isn't much more than that as far as error message detail goes. – Ryan J. McDonough Nov 13 '14 at 12:40
  • Almost certainly a memory issue. seems like some resources are not being freed up. Over time they seem to build up till it crashes out with this error and you cant even get into Jenkins. using `service jenkins restart` seems to fix the problem but this is a short term solution to a long term problem. – Joe Lloyd Aug 14 '15 at 13:52

5 Answers5

50

I was running into the same behavior on our build server. The error is IMHO not associated with the maven memory settings (i.e. MAVEN_OPTS) but rather with the memory of the underlying (Linux) machine itself (which Jenkins runs on).

The (rejected) Jenkins issue https://jenkins-ci.org/issue/12035 gives more detail on this matter:

For reference the status code 137 (128 + 9) typically means (can differ between flavours of unix). That the process was terminated by receipt of a signal. In this case signal 9 which is SIGKILL and unblockable kill.

If this is the case the underlying machine/OS needs more virtual memory. This can be added by either adding physical memory or swap space as appropriate.

You should try to increase the virtual memory of your machine.

Note:
This also explains why a Jenkins restart (temporarily) fixes the issue.

boskoop
  • 1,157
  • 1
  • 10
  • 17
  • 4
    Please note: I answered this (over 1 year old) question because I ran into the same problem and it turned up being the first hit on Google. – boskoop Dec 08 '15 at 18:02
  • 1
    We were running Jenkins as a daemon and found adjusting the "-Xmx" startup param solved the problem -- "/usr/bin/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -XX:MaxPermSize=512m -Xmx8192m -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --ajp13Port=-1 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20" – Dave Jun 28 '17 at 13:59
8

I believe you should increase the values of the memory settings - in MAVEN_OPTS on the Jenkins machine, e.g.

MAVEN_OPTS=-Xmx1.5G -XX:MaxPermSize=0.7G
Mateva
  • 786
  • 1
  • 8
  • 27
  • You can attach a JVM Monitor(or similar software) to monitor if you actually have a memory leak. – Mateva Jul 28 '14 at 12:03
  • I am using VisualVM can you please guide me on how to detect a memeory leak on that. – user2822319 Jul 28 '14 at 12:05
  • ... I see this now... You make a heap dump, then you check classes - see what kind of objects are there that are hanging and look for which classes hold them into memory. – Mateva Mar 25 '16 at 13:36
3

If you machine has at least 2 processors and 4GB memory, your JVM will not only grab 1GB at startup but will turn -server mode, meaning memory will be retained for performance sake (source). If you have few JVMs running at the same time (several application components, maven builds etc.) you can easily get into low memory. And one of you JVM may be killed by Linux OOM Killer because you are low on resources on the machine.

Reduce memory footprint of your process which is directly impacted by jvm default Xmx, which most probably is far from what jvm actually need.

Give it additional java command line options

-Xmx256m -XX:MaxPermSize=512m

or configure system variable

MAVEN_OPTS=-Xmx256m -XX:MaxPermSize=512m

MaxPermSize have no use for java 8+

Mike
  • 20,010
  • 25
  • 97
  • 140
1

When running Maven via Jenkins I got this error:

ERROR: Maven JVM terminated unexpectedly with exit code 137

I accidently put an "and" in the MAVEN_OPTS parameter:

-Xmx1024m and -Xms1024m

Then, I got this error:

Error: Could not find or load main class and
ERROR: Failed to launch Maven. Exit code - 1

After, removing the 'and', I reran Jenkins and received this error:

java.lang.OutOfMemoryError: Java heap space

Finally, I increased the memory using Global MAVEN_OPTS:

-Xmx4096m -Xms4096m

This fixed the problem. So, this seems to be related to memory. However, it could be a machine/VM related issue (as @boskoop stated above) or a container issue (if JVM is run through Jenkins/Docker/etc).

Scott Izu
  • 2,229
  • 25
  • 12
0

I ran into the same error code. This error code does indeed seem related to JVM resource constraints in the Jenkins environment. I would suggest re-running the build a second time after the error occurs to see if you can get additional/different output, though this will certainly depend on which part of the build caused the resource issues (in my cause it was a Maven downloads). 

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000fb7cb000, 7331840, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 7331840 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /vagrant/args4java/hs_err_pid10470.log
ERROR: Maven JVM terminated unexpectedly with exit code 1
Finished: FAILURE
DTS
  • 315
  • 5
  • 12