Correct way to System.setErr() for multiple threads?

Question

So I've got a Java application using a fixed thread pool to execute some code. This code includes use of a third-party library that outputs to System.err. When I had this chunk of code executing single-threaded, I "redirected" System.err to a PrintStream that ultimately ended up printing out to log4j logs. Basically it looked like this:

 PrintStream oldErr = System.err;
 System.setErr(new PrintStream(/* custom logging stream here */));

 try
 {
     // do computationally expensive task here
 }
 finally
 {
     System.setErr(oldErr);
 }

That worked as expected. The output printed to the logfiles instead of the console, and I could remove the output entirely by changing my log4j configuration. Perfect.

When I started to add multithreading I did a little research and came across this SO question: In a multithreaded Java program, does each thread have its own copy of System.out?, which implied that I should do System.setErr() once before firing up my thread pool and I'd be all set. Only that doesn't appear to be the case. My code looks like this now:

 PrintStream oldErr = System.err;
 System.setErr(new PrintStream(/* custom logging stream here */));

 ExecutorService threadPool = Executors.newFixedThreadPool(maxThreadCount);

 try
 {
     // shove computationally expensive task off to a thread
     //   using threadPool.execute()
 }
 finally
 {
     System.setErr(oldErr);
 }

However, calling System.setErr() before firing up the thread pool has zero effect. The threads all print their output to System.err as if I had not made the call at all. Drat!

I also tried having the threads call System.setErr() while executing their tasks, but there were some clear race condition issues with that--intermittent output to the console, and a general feeling of dirtiness. In one test run it even looked like they deadlocked.

Am I missing something simple? FWIW, here are my JVM details:

java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b07)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b17, mixed mode)

Thanks.

EDIT: My issue was basically solved by the accepted answer, but for sake of completeness I wanted to add that I could not solve my particular case using Futures and get(). My individual tasks consume a lot of RAM but vary in duration, so I had to go with a small blocking queue as outlined in the answer to Java: ExecutorService that blocks on submission after a certain queue size. I basically fell into the trap of thinking that the defaults in newFixedThreadPool() would work for my situation when in reality they did not, and Brian's answer helped expose those faulty assumptions. Thanks Brian!

Answer 1

The following SSCCE does exactly what you want and are expecting:

public class App
{
    public static void main(String[] args) throws FileNotFoundException, InterruptedException, ExecutionException 
    {
        PrintStream oldErr = System.err;
        System.setErr(new PrintStream(new File("test")));

        ExecutorService threadPool = Executors.newFixedThreadPool(5);
        List<Future<?>> fList = new LinkedList<Future<?>>();
        for (int i = 0; i < 5; i++)
        {
            fList.add(threadPool.submit(new Runnable() {

                @Override
                public void run()
                {
                    System.err.println("This is to the err stream");
                }

            }));
        }

        for (Future<?> f : fList)
        {
            f.get();
        }

        threadPool.shutdown();

        System.setErr(oldErr);
        System.err.println("This is to the err stream");
    }
}

After running this example, the file "test" contains 5 lines, each saying "This is to the err stream" and at the end one additional line is printed to the console.

The code snippet you show is calling System.setErr(oldErr); in the finally block ... have you waited for all your tasks to complete in that try block?

If not it would explain the statement you make in the comments that "I would see fragments of output printed to System.err and fragments printed to my logfile" ... that's pretty much what you'd expect to happen. .submit() is a non-blocking call that returns a Future; you're resetting the stream while your tasks are running.