21

It looks like 

 MemoryError: PermGen space
 java.lang.OutOfMemoryError: PermGen space

is a common problem. You can Increase the size of your perm space, but after 100 or 200 redeploys it will be full. Tracking ClassLoader memory leaks is nearly impossible.

What are your methods for Tomcat (or another simple servlet container - Jetty?) on production server? Is server restart after each deploy a solution?

Do you use one Tomcat for many applications ?

Maybe I should use many Jetty servers on different ports (or an embedded Jetty) and do undeploy/restart/deploy each time ?

Pascal Thivent
  • 562,542
  • 136
  • 1,062
  • 1,124
Piotr Gwiazda
  • 12,080
  • 13
  • 60
  • 91

7 Answers7

6

I gave up on using the tomcat manager and now always shutdown tomcat to redeploy.

We run two tomcats on the same server and use apache webserver with mod_proxy_ajp so users can access both apps via the same port 80. This is nice also because the users see the apache Service Unavailable page when the tomcat is down.

Adam Butler
  • 3,023
  • 5
  • 35
  • 40
  • 1
    We use two tomcats on different ports behind HAProxy to provide blue-green deployment solution. "Blue" server is shut down when "Green" is up and running, HAProxy switches to "Green" then. Next deploy goes to "Blue" then. – Piotr Gwiazda Apr 25 '12 at 07:56
3

You can try adding these Java options:

-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

This enables garbage collection in PermGen space (off by default) and allows the GC to unload classes. In addition you should use the -XX:PermSize=64m -XX:MaxPermSize=128m mentioned elsewhere to increase the amount of PermGen available.

werkshy
  • 1,605
  • 1
  • 13
  • 22
Soundlink
  • 3,915
  • 2
  • 28
  • 36
2

Yes indeed, this is a problem. We're running three web apps on a Tomcat server: No. 1 uses a web application framework, Hibernate and many other JARs, no. 2 uses Hibernate and a few JARs and no. 3 is basically a very simple JSP application.

When we deploy no. 1, we always restart Tomcat. Otherwise a PermGen space error will soon bite us. No. 2 can sometimes be deployed without problem, but since it often changes when no. 1 does as well, a restart is scheduled anyway. No. 3 poses no problem at all and can be deployed as often as needed without problem.

So, yes, we usually restart Tomcat. But we're also looking forward to Tomcat 7, which is supposed to handle many memory / class loader problems that are burried into different third-party JARs and frameworks.

Codo
  • 75,595
  • 17
  • 168
  • 206
2

PermGen switches in HotSpot only delay the problem, and eventually you will get the OutOfMemoryError anyway.

We have had this problem a long time, and the only solution I've found so far is to use JRockit instead. It doesn't have a PermGen, so the problem just disappears. We are evaluating it on our test servers now, and we haven't had one PermGen issue since the switch. I also tried redeploying more than 20 times on my local machine with an app that gets this error on first redeploy, and everything chugged along beautifully.

JRockit is meant to be integrated into OpenJDK, so maybe this problem will go away for stock Java too in the future.

http://www.oracle.com/technetwork/middleware/jrockit/overview/index.html

And it's free, under the same license as HotSpot:

https://blogs.oracle.com/henrik/entry/jrockit_is_now_free_and

Niklas Herder
  • 111
  • 7
  • I thought JRockit keeps class definitions on heap. Don't your heap grow after each redeploy? What does "JRockit Mission Control" show about heap size? – Piotr Gwiazda Feb 22 '12 at 09:14
  • The heap stays at the same size after each redeploy, as far as I can see. I tried again with about 10 redeploys right now. – Niklas Herder Feb 23 '12 at 14:08
1

Which version of Tomcat are you using? Tomcat 7 and 6.0.30 have many features to avoid these leaks, or at least warn you about their cause.

This presentation by Mark Thomas of SpringSource (and longtime Tomcat committer) on this subject is very interesting.

Daniel Serodio
  • 4,229
  • 5
  • 37
  • 33
  • The question was asked in 2010. Tomcat 7 was in plans then. Our solution was an application that shut down tomcat, copies old WAR to backup dir, downloads new WAR from Hudson, starts Tomcat. We use two tomcats and blue-green deployment pattern. – Piotr Gwiazda Apr 25 '12 at 07:52
1

Just of reference, there is a new version of Plumbr tool, that can monitor and detect Permanent Generation leaks as well.

Nikem
  • 5,716
  • 3
  • 32
  • 59
1

You should enable PermGen garbage collection. By default Hotspot VM does NOT collect PermGen garbage, which means all loaded class files remain in memory forever. Every new deployment loads a new set of class files which means you eventually run out of PermGen space.

rustyx
  • 80,671
  • 25
  • 200
  • 267
  • As noted by @Soundlink, the parameters to turn on GC in PermGen are "-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled". On Ubuntu you would add these to JAVA_OPTS in /etc/default/tomcat6. – werkshy Jun 28 '11 at 21:53