1

I am dealing with below OutOfMemory exception in WAS 6.1.

Exception in thread "UnitHoldingsPolicySummary" java.lang.OutOfMemoryError: unable to create new native thread.

I have done a lot of research on this to prevent this. After Googling, I have found that, this happens when the Native memory gets exhausted due to creation of lots of threads concurrently.

Now, after analysing the below logs, we can figure out that, inside the application, the threads are created explicitely, which I read is a very very bad practice. (Can experts please confirm this?)

07/07/14 08:50:38:165 BST] 0000142c SystemErr     R Exception in thread "xxxxxx" java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:574)
        at com.fp.sv.controller.business.thread.xxxxxxxxxexecute(Unknown Source)
        at com.fp.sv.controller.business.thread.xxxxxxxxx.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:595)

I am more into WAS administration and doesn't possess much knowledge on Java and thread creation in Java. Now I need to discuss this with developer, but before that I want to be 100% confirmed that my findings are correct and developers should correct the code by not explicitely creating the threads.

What all things that I need to check on application server side before blaming this on code?

On solaris, I am firing the command pmap -x 9547|grep -i stack|wc -l to check how many threads are getting created on that instance of time. I could see during the 'OutOfMemory' issue, this number is very high.

Could you please confirm whether this command is the good way to checknumber of threads currently active?

Editing the question with my latest findings

Also, when this issue happens, at the same one of the MQ queue gets piled up as WAS doesn't pick up the messages from the queue. I could see below error in the application specific logs.

Non recoverable Exception detected whilst connecting to queue manager or response queue Underlying reason = MQJE001: Completion Code 2, Reason 2102

Can this issue related to MQ as well?Which in turn causes OutOfMemory issue?

Regards, Rahul

ﾤ ﾡ
  • 5,669
  • 6
  • 43
  • 57
user2607367
  • 225
  • 4
  • 25

2 Answers2

1

There are different possibilities of implementing a threading system for a virtual machine. The two extreme forms are:

  1. Green threads: All Java Thread instances are managed within one native OS thread. This can cause problems if a method blocks within a native invocation what makes this implementation complex. In the end, implementers need to introduce renegade threads for holding native locks to overcome such limitations.
  2. Native threads: Each Java Thread instance is backed by a native OS thread.

For the named limitations of green threads, all modern JVM implementations, including HotSpot, choose the later implementation. This implies that the OS needs to reserve some memory for each created thread. Also, there is some runtime overhead for creating such a thread as it needs direct interaction with the underlying OS. At some point, these costs accumulate and the OS refuses the creation of new threads to prevent the stability of your overall system.

Threads should therefore be pooled for resue. Object pooling is normally considered bad practice as many programers used it to ease the JVM's garbage collector. This is not longer useful as modern garbage collectors are optimized for handling short-living objects. Today, by pooling objects you might in contrary slow down your system. However, if an object is backed by costly native resources (as a Thread), pooling is still a recommended practice. Look into the ExecutorService for a canonical way of pooling threads in Java.

In general, consider that context switches of threads are expensive. You should not create a new thread for small tasks, this will slow your application down. Rather make your application less concurrent. You only have a number of cores which can work concurrently in the first place, creating more threads than your (non-virtual) cores will not improve runtime performance. Are you implementing some sort of divide-and-conquer algorithm? Look into Java's ForkJoinPool.

Rafael Winterhalter
  • 42,759
  • 13
  • 108
  • 192
0

Yes, it's a bad practice. Normally, you don't manage threads inside a Java EE server. By "normally" I mean "while developing business applications".

According to http://www.oracle.com/technetwork/java/restrictions-142267.html:

Why is thread creation and management disallowed?

The EJB specification assigns to the EJB container the responsibility for managing threads. Allowing enterprise bean instances to create and manage threads would interfere with the container's ability to control its components' lifecycle. Thread management is not a business function, it is an implementation detail, and is typically complicated and platform-specific. Letting the container manage threads relieves the enterprise bean developer of dealing with threading issues. Multithreaded applications are still possible, but control of multithreading is located in the container, not in the enterprise bean.

However, I don't think your logs demonstrate that threads are being created explicitly. If you want to be 100% sure, decompile the deployables and look at the code in those lines.

Also take a look at this:

"java.lang.OutOfMemoryError : unable to create new native Thread"

And this:

https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread

Concerning the number of threads used by your app, I'd try to use a monitoring tool like JConsole, or VisualVm.

Community
  • 1
  • 1
Andres
  • 10,561
  • 4
  • 45
  • 63
  • Thanks! does the logs which i have posted confirms that the threads are being created explicitely? – user2607367 Jul 07 '14 at 13:54
  • 1
    *Normally, you don't manage threads inside a JEE server.* Of course you do. Not all threads, but you can integrate your own code which can of course manage its own threads for its tasks. – Rafael Winterhalter Jul 07 '14 at 14:07
  • @raphw, please show me an article, book or paper that recommends the explicit creation and management of threads in a JEE server. – Andres Jul 07 '14 at 14:18
  • @Andres Show me one that forbids it. If I write a framework that requires a thread for its internal work, would I not be allowed to use this framework in JavaEE? Akka, just to name one. And what would guarantee that I would not need to write custom code that requires threads for equally legal reasons? You might think about a servlet container's dispatch threads. Those are JEE specific and you should not interact with them directly, but you might in general create threads. – Rafael Winterhalter Jul 07 '14 at 14:22
  • @Andress: I think below lines in the logs do suggestes that the thread 'xxx are being created explicitely? isn't it? `R Exception in thread "xxxxxx" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:574) at com.fp.sv.controller.business.thread.xxxxxxxxxexecute(Unknown Source) at com.fp.sv.controller.business.thread.xxxxxxxxx.run(Unknown Source) at java.lang.Thread.run(Thread.java:595) ` – user2607367 Jul 07 '14 at 14:40
  • The log does suggest this, it is not your JEE container specific threads that cause the problems. It is the application that creates the threads. Without further information, it is difficult to tell you more. Is it an obfuscated third party app? Maybe the obfuscation broke something. – Rafael Winterhalter Jul 07 '14 at 15:22
  • 2
    @Andres Nice revenge down vote, that has class. I believe that your answer is incorrect and I explained you why: The statement *you don't manage threads inside a JEE server* is simply false. With multiple processors available, it is not a problem to write multi-threaded applications and deploy them into JEE containers. This is not the problem, the OP is having. The problem is that threads are created explicitly instead of using a pool. The solution is to manage the threads, not to get rid of them. – Rafael Winterhalter Jul 07 '14 at 15:41
  • The statement "you don't manage threads inside a JEE server" is simply false. BUT, the statement "**Normally**, you don't manage threads inside a JEE server" is true. And I posted a link (from Oracle) to support my opinion. – Andres Jul 07 '14 at 15:45
  • 1
    It says that an EJB container *places certain restrictions on the components it manages* and than lists these restrictions. What makes you think that the created threads relate to such a component? Does neither justify a revenge down vote, btw. – Rafael Winterhalter Jul 07 '14 at 15:47
  • A JEE server normally includes developer documentation outlining how to properly get hold of a thread / execute a runnable in a thread inside the JEE server. Both WAS and JBoss have this documentaion, but as a developer it is a pain to follow these guidelines (complex, takes time). Which makes a shotcut very attractive. I found that taking the shortcut introduces subtle bugs which in turn make the component unstable (over time). It is not unlikely something like that happened here. – vanOekel Jul 07 '14 at 18:08
  • 1
    Java EE has a few explicit mechanisms available to create *managed* threads. Managed threads are those that preserve some context, like the Java EE component namespaces, an authenticated identity (if any), transactional status, etc. I collected a few links about that topic here: https://javaee7.zeef.com/arjan.tijms#block_237 But if you don't need this context (e.g. do some raw computation, or bootstrap EJB beans by looking them up from the java:global space) AND you manage your threads in a pool than there's IMHO nothing wrong with handling your own threads. – Arjan Tijms Jul 07 '14 at 18:40
  • Hello, I have added few more information about the output of pmap command. Could you please help me to confirm whether my understadning is correct? – user2607367 Jul 08 '14 at 08:14
  • The command is a good way for determining this. However, the JVM error should normally be good enough. I am pretty sure that the conclusion is pretty certain that it is an application error. A question: Is the application obfuscated, i.e. is it a commercial application without open sources? To me, it looks like the it is as the stack is so small and the names rather uncommon. This obfuscation might have caused problems. – Rafael Winterhalter Jul 08 '14 at 09:32
  • I have updated the quetion with my lates finding, where I have mentioned about mq error code. – user2607367 Jul 08 '14 at 11:49