OutOfMemory error with FixedThreadPool and ExecutorCompletionService

Question

I working on app what should get list of users from db and update thie details from directory (ldap or AD). I what do this procedure on multicore machine so I created this app (code below). I'm using CompletionService and getting the results in Future object.

After some time, I'm getting out of memory error with 'unable to create new native thread' message. In task manager I see that app created enormous amount of threads but I asked create fixed thread pool with size equals amount of my processors.

What is wrong with my code?

class CheckGroupMembership {
public static void main(String[] args) throws Exception {

    final ExecutorService executor = Executors.newFixedThreadPool(**Runtime.getRuntime().availableProcessors()**);

    CompletionService<LdapPerson> completionService =
        new ExecutorCompletionService(executor)<LdapPerson>(executor);

    final int limit = 2000;

    DocumentService service1 = new DocumentService();
    List<String> userNamesList = service1.getUsersListFromDB(limit);

    List<LdapPerson> ldapPersonList = new ArrayList() <LdapPerson> (userNamesList.size());
    LdapPerson person;

    for (String userName : userNamesList) {
        completionService.submit(new GetUsersDLTask(userName));
    }

    try {
        for (int i = 0, n = userNamesList.size(); i < n; i++) {
            Future<LdapPerson> f = completionService.take();
            person = f.get();
            ldapPersonList.add(person);
        }
    } catch (InterruptedException e) {

        System.out.println("InterruptedException error:" + e.getMessage());
    } catch (Exception e) {
        System.out.println(e.getMessage());
    }
    System.exit(0);
}
}

ERROR CheckGroupMembership:85 - java.lang.OutOfMemoryError: unable to create new native thread java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create new native thread at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83

GetuserDLs task

public class GetUsersDLTask implements Callable<LdapPerson> {
private String userName;

public GetUsersDLTask(String u) {
    this.userName = u;
}

@Override
public LdapPerson call() throws Exception {
    LdapService service = new LdapService();
    return service.getUsersDLs(userName);
}

}

Request a thread dump from your JVM. What are the names of the threads? Does LdapService create a thread? — Emil Sit, Oct 27 '11 at 16:20
LdapService doesn't create any threads. I'm using standar javax.naming.* stuff — Vik Gamov, Oct 27 '11 at 16:52
What is in the thread dump from the JVM prior to the OOM? What are the threads named? — Emil Sit, Oct 27 '11 at 16:59

score 3 · Accepted Answer · answered Oct 27 '11 at 17:36

I am having a hard time believing you are not creating a Thread in GetUsersDLTask (or at least it's service Object). If you look at your stacktrace, the Exception is being thrown from the Future's get() method. The only way this exception get's set is after the Executor invokes Callabale.call(). Any throwable that occurs in the call() method will be set in the Future's internal exception field

For example:

Thread Pool: 
    Thread-1
      invoke call()
        call() 
          Create Thread
            throw OutOfMemoryError 
         propogate error to Thread pool
      set exception

Otherwise, this exception would be occurring when you submit the ask to thread pool, not when you get from the future.

Yes, I beleive you're right.In GetUsersDLTask I created new instance of LdapService which creates InitialContext object each and every time (and it's create new thread for socket communication with ldap). So your assumption on mnemocode is correct. Thanks for advice! — Vik Gamov, Oct 27 '11 at 18:49

score 1 · Answer 2 · edited Oct 27 '11 at 17:06

1

Executors.newFixedThreadPool will accept the submission of many tasks, but will only execute the number of threads that you allow. So if you have a fixed pool of 2 threads, but you submit 50 tasks, the 48 other tasks are queued internally on the executor and are run as the executing threads finish tasks. Seems like you need to limit the number of threads you are spawning within your code.

Edit: check out http://download.oracle.com/javase/6/docs/api/java/util/concurrent/Executors.html#newFixedThreadPool(int)

edited Oct 27 '11 at 17:06

Community

1
1

answered Oct 27 '11 at 16:18

Noah

1,966
1
14
29

But did executor instantiates rest 48 threads at the time? How I can control thread spawning? – Vik Gamov Oct 27 '11 at 16:57
1

As the proposed edit indicates, you submit tasks, and the thread pool manages the threads. – Emil Sit Oct 27 '11 at 16:59
1

huh, why the down vote? You are running out of memory because you are spawning too many threads - the executor will take as many as you give it, but will only execute the number provided in the fixed pool. You can control thread spawning because you are the one submitting the threads to the executor; do you really need a new thread for each userName? Perhaps parse them up into groups of 100's or so and submit that in a thread. – Noah Oct 27 '11 at 17:03
The question and the comment were two separate thoughts. No worries :) – Noah Oct 27 '11 at 17:13
According to this javadoc example http://download.oracle.com/javase/6/docs/api/java/util/concurrent/ExecutorCompletionService.html I do not do anything forbidden with ECS. I'm not creating threads explicitly. – Vik Gamov Oct 27 '11 at 17:14
Yes - the creation of the threads/tasks is being done correctly. You are just creating a ton of them - one per username. Do you need so many? – Noah Oct 27 '11 at 17:15
GetUsersDLTask implements Callable. Callable is like Runnable; a new Callable when 'submitted' to the executor will create a new thread (and return the FtureTask) and potentially be added to the queue. The CompletionService takes in the instantiated executor; its just a wrapper that makes use of the threading behind the scenes. IN this example, every submitted Callable will be converted to a thread by the CompletionService by means of the Executor. – Noah Oct 27 '11 at 17:42
1

@sqrv That's not true. When you submit a Callable/Runnable it will be placed on the ES work queue. If the thread pool is fixed, then each Runnable/Callable will only be used when a thread is available and can poll off the queue. If you have a `Executors.newCachedThreadPool` then yes, each Callable submitted would spawn a new thread if the thread isn't idling. This isn't that case – John Vint Oct 27 '11 at 18:17
Ah! I'm mixing the term 'Task' and 'Thread' up as the same, where as in reality, there is a functional difference. Thanks for the clarification John. – Noah Oct 27 '11 at 18:30

score 0 · Answer 3 · answered Oct 27 '11 at 17:25

0

Did you verify the number of threads created in the fixed pool. Maybe the # of available processors is coming out as too big.

answered Oct 27 '11 at 17:25

Dave

5,133
21
27

OutOfMemory error with FixedThreadPool and ExecutorCompletionService

3 Answers3