4

I'm a bit of a novice when it comes to JAVA applications, but have been involved in developing a fairly complex JAVA(8) app that requires multi-threading. Myself and another developer have kept running into a problem where the app keeps running out of memory after running for a while.

At first we gave the application 64GB of memory, but after a few hours it'd run out of memory, crash and restart. Only to keep doing it over and over. Context; The application takes messages from a messaging system (ActiveMQ) and from the message's meta has to build an XML file by calling various data sources for values. There could be literally millions of messages that need to be processed, so we developed a multi-threading system, each thread deal with a message - and gave the application 40 threads.

However, as it keeps taking messages the overall memory consumption goes up and up over time. I feel like the garbage collector isn't being utilized by us correctly?

So at the moment we have one parent thread:

(new Thread(new ReportMessageConsumer(config, ""))).start();

Then within the ReportMessageConsumer we have X number of threads setup, so this would be 40 in our current setup. So this would be all under this one group. Once the XML has been built and the threads done with how do we effectively kill the thread and enforce the Garbage collector to free that memory, so that we can then create a new clean thread to pick up another message?

Simon Nicholls
  • 93
  • 2
  • 10
  • A thread dies shortly after its `run()` method either returns or throws an exception. If your code hasn't kept any reference to the `Thread` object (and your example clearly does not keep one), then the `Thread` object, and all of the objects referenced by the thread's stack should immediately become elgible to be reclaimed by the GC. – Solomon Slow Jun 30 '16 at 12:59
  • 2
    Don't use threads directly if you want to kill one and start another to process "another message": use executors. – Andy Turner Jun 30 '16 at 12:59
  • What is the exact OutOfMemoryError are you seeing? Is it heap, stack, gc overhead, unable to create native threads? – John Vint Jun 30 '16 at 13:23
  • It's the one where it has the attempted byte allocation in the message. So I think the unable to create native threads ? – Simon Nicholls Jun 30 '16 at 13:44
  • @AndyTurner Yeah, answer below has helped me understand that! – Simon Nicholls Jun 30 '16 at 13:44
  • @SimonNicholls If you were in fact seeing the `cannot create native thread` message then that means you were simply creating too many threads, on the order of thousands (to 10s of thousands). That would explain your OOM error and will be resolved by using an `ExecutorService` so long as you DONT shut it down after initial use and create a new one. – John Vint Jun 30 '16 at 18:09

1 Answers1

7

I feel like the garbage collector isn't being utilized by us correctly?

That is not the problem. The best thing you can do is to let the GC do its thing without any interference. Don't try to force the GC to run. It is rarely helpful, and often bad for performance.

The real problem is that you have a memory leak. It may be happening because you are getting more and more threads ... or it may be something else.

I would recommend the following:

  1. Rewrite you code so that it uses a ExecutorService to manage a bounded pool of threads, and a queue of tasks to be run on those threads. Look at the javadocs for a simple example.

    Using a thread pool is likely to improve your application's overall performance. Creating a thread (i.e. Thread.start()) is rather expensive in Java.

    (And don't shut down the pool as a way to ensure that a batch of work has completed. That is bad for performance. The simple way to do that is to submit the batch using invokeAll; see ExecutorService, how to wait for all tasks to finish.)

  2. If that doesn't cure your leak, then use a memory profiling tool to find out how / why your application is leaking memory. There are lots of StackOverflow Q&A's on how to do this. For example:

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
  • Thanks for this, does something like: `executorService = (ThreadPoolExecutor) Executors.newFixedThreadPool(maxThread); while (true) { if (executorService.getActiveCount() < maxThread) { executorService.execute(this::processMessage); } }` Seem like a reasonable way of managing the Executors with the limits of the number of threads ? – Simon Nicholls Jun 30 '16 at 13:31
  • 1
    A thread pool, like the ExecutorService, should have a fixed number of threads which is backed by a queue. So, if you assign 10 threads to the thread pool it will never have more threads, but if all threads are busy the `processMessage` will be queued for eventual execution. In other words, you need not worry about `getActiveCount` or any thread management – John Vint Jun 30 '16 at 13:34
  • Thanks, that makes things simpler! So if we were still running into memory leaks at that point, it's because something within the executorService is being stored into memory still ? I guess when we call executorService.shutdown() at the end of each message that's been processed it should empty everything from it using GC ? – Simon Nicholls Jun 30 '16 at 13:42
  • 1
    @Simon Nicholls: **don’t** invoke `shutdown()` at the end of each message, that’s perverting the entire purpose of the executor service. If your application is still running out of memory after limiting the number of threads, it’s *your* code, not the executor service which causes the problem, i.e. you are storing things into global variables inside the jobs. – Holger Jun 30 '16 at 16:32