6

I've been having a pretty serious production issue and I was hoping that someone could help out.

I am using Spring 3.1.0 and have been impressed by its capabilities.

During development, everything was fine. Unfortunately, upon deploying to production, we came up against some issues.

We are getting PermGen space out of memory errors.

On our dev/test box, we have the following settings: -Xms2048m -Xmx2048m -XX:PermSize=512m -XX:MaxPermSize=1024m (please note that we are not confident that these are correct settings)

On our production box, we have the following settings: -Xms3056m -Xmx3056m -XX:PermSize=128m -XX:MaxPermSize=512m (again, please note that we are not confident that these are correct settings)

Admittedly, the production environment is quite different in that it has a lot more applications running on it.

According to what I've read here, we should set the max and min to the same value and the maxpermsize to 1/4 of either one of those.

I have read in a number of locations (including here) that it may be unavoidable to restart a server due to leaks. Does anybody know if this is true?

Also, does anybody know how much memory would typically be needed by the Spring framework? Upon deployment of the application, I find that the amount of memory used increases by approximately 345 MB. I understand that a number of classes need to be instantiated on startup of the application but this seems like a lot. Do I have something setup incorrectly? We are only using Spring JDBC at the moment. Is there some way to limit the framework to those classes only?

Our code base is rather large and I understand that JAXB can cause serious problems.

I have seen recommendations of using

-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

here and here. Are there any possible negative side effects to using these arguments?

So, to summarise:

  1. Are the settings we have correct and, if not, what should they be?

  2. Are server restarts from time to time unavoidable with the Spring framework?

  3. How much memory should Spring be taking up? Is there a way to limit the classes used?

  4. Are there any possible side effects to using -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled ?

Any and all help would be greatly appreciated. Thanks in advance.

The stack trace is below (there is more but I couldn't post it all):

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter': Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter]: Constructor threw exception; nested exception is java.lang.OutOfMemoryError: PermGen space
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:997)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:943)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:286)
    at org.springframework.web.servlet.DispatcherServlet.createDefaultStrategy(DispatcherServlet.java:788)
    at org.springframework.web.servlet.DispatcherServlet.getDefaultStrategies(DispatcherServlet.java:757)
    at org.springframework.web.servlet.DispatcherServlet.initHandlerAdapters(DispatcherServlet.java:565)
    at org.springframework.web.servlet.DispatcherServlet.initStrategies(DispatcherServlet.java:422)
    at org.springframework.web.servlet.DispatcherServlet.onRefresh(DispatcherServlet.java:410)
    at org.springframework.web.servlet.FrameworkServlet.onApplicationEvent(FrameworkServlet.java:752)
    at org.springframework.web.servlet.FrameworkServlet$ContextRefreshListener.onApplicationEvent(FrameworkServlet.java:989)
    at org.springframework.web.servlet.FrameworkServlet$ContextRefreshListener.onApplicationEvent(FrameworkServlet.java:1)
    at org.springframework.context.event.GenericApplicationListenerAdapter.onApplicationEvent(GenericApplicationListenerAdapter.java:51)
    at org.springframework.context.event.SourceFilteringListener.onApplicationEventInternal(SourceFilteringListener.java:97)
    at org.springframework.context.event.SourceFilteringListener.onApplicationEvent(SourceFilteringListener.java:68)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:97)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:324)
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:929)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:467)
    at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:631)
    at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:588)
    at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:645)
    at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:508)
    at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:449)
    at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:133)
    at javax.servlet.GenericServlet.init(GenericServlet.java:241)
    at weblogic.servlet.internal.StubSecurityHelper$ServletInitAction.run(StubSecurityHelper.java:283)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
    at weblogic.servlet.internal.StubSecurityHelper.createServlet(StubSecurityHelper.java:64)
    at weblogic.servlet.internal.StubLifecycleHelper.createOneInstance(StubLifecycleHelper.java:58)
    at weblogic.servlet.internal.StubLifecycleHelper.<init>(StubLifecycleHelper.java:48)
    at weblogic.servlet.internal.ServletStubImpl.prepareServlet(ServletStubImpl.java:539)
    at weblogic.servlet.internal.WebAppServletContext.preloadServlet(WebAppServletContext.java:1976)
    at weblogic.servlet.internal.WebAppServletContext.loadServletsOnStartup(WebAppServletContext.java:1950)
    at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAppServletContext.java:1869)
    at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletContext.java:3126)
    at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1512)
    at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:486)
    at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425)
    at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
    at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119)
    at weblogic.application.internal.flow.ScopedModuleDriver.start(ScopedModuleDriver.java:200)
    at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:247)
    at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425)
    at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
    at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119)
    at weblogic.application.internal.flow.StartModulesFlow.activate(StartModulesFlow.java:27)
    at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:1267)
    at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
    at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:409)
    at weblogic.application.internal.EarDeployment.activate(EarDeployment.java:58)
    at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:161)
    at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:79)
    at weblogic.deploy.internal.targetserver.operations.AbstractOperation.activate(AbstractOperation.java:569)
    at weblogic.deploy.internal.targetserver.operations.ActivateOperation.activateDeployment(ActivateOperation.java:150)
    at weblogic.deploy.internal.targetserver.operations.ActivateOperation.doCommit(ActivateOperation.java:116)
    at weblogic.deploy.internal.targetserver.operations.StartOperation.doCommit(StartOperation.java:143)
    at weblogic.deploy.internal.targetserver.operations.AbstractOperation.commit(AbstractOperation.java:323)
    at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentCommit(DeploymentManager.java:844)
    at weblogic.deploy.internal.targetserver.DeploymentManager.activateDeploymentList(DeploymentManager.java:1253)
    at weblogic.deploy.internal.targetserver.DeploymentManager.handleCommit(DeploymentManager.java:440)
Community
  • 1
  • 1
the_new_mr
  • 3,476
  • 5
  • 42
  • 57
  • 1
    Re:"2.Are server restarts from time to time unavoidable with the Spring framework?" - Nope - we use Spring 3.0.4 in our prod env and we dont have any memory issues. Could it be one of the other libraries you're using? We had a problem with one of the weblogic jars conflicting with xmlbeans and leading to OOM – JoseK Aug 14 '12 at 06:31
  • 1
    We had an OOM with PermGen with initial of 128 so we set it to 256 which kept us fine. I can see on dev you've already given 512, can you do an initial of 256 on Prod. The other track is to examine the PermGen contents like so http://stackoverflow.com/questions/4080010/how-to-dump-permgen and http://stackoverflow.com/questions/2842982/how-to-analyze-permgen-contents – JoseK Aug 14 '12 at 06:40
  • I did some analysis with JVisualVM but wasn't able to see the objects in the perm space. We tried changing the production settings to 3 GB for heap size and 1 GB for the max perm size and this *seems* to have fixed the problem for now. I shall definitely look into links you posted as this could be useful in the future. Thanks. – the_new_mr Aug 14 '12 at 14:48
  • 1
    Ok, if you're on 32 bit OS & JVM 3 GB heap is the tops and adding 1 Gb for perm is additional JVM memory space addressing - imo a bit risky. if you're on 62 bit OS & JVM then all cool – JoseK Aug 16 '12 at 06:11
  • It's a 64 bit system so good news. Thanks! – the_new_mr Aug 20 '12 at 16:29

2 Answers2

4

In order to see the permGen VM, you can look at MemoryPool MBean in JConsole. Also, you can add the following JVM arguments (in case you don't have already) 

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/msp/log

for heapdumps collection.

-XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/opt/msp/log/gc.log`

for GC related information.

This will help in understanding Garbage Collection pattern and may guide you in coming up with a better limits.

raoadnan
  • 293
  • 2
  • 10
0

It is hard to tell how much memory is optimal for a Production environment. The -Xms need not be equal(or as high as -Xmx). You can set it to a lower value but increase the value of -XX:PermSize. Also to arrive at what values are optimal you should try and benchmark to see what gives optimal results.

Naidu Ypvs
  • 161
  • 10