How to test task performance, using multitheading?

Question

I have some exercises, and one of them refers to concurrency. This theme is new for me, however I spent 6 hours and finally solve my problem. But my knowledge of corresponding API is poor, so I need advice: is my solution correct or may be there is more appropriate way.

So, I have to implement next interface:

public interface PerformanceTester {    
    /**
     * Runs a performance test of the given task.
     * @param task which task to do performance tests on
     * @param executionCount how many times the task should be executed in total
     * @param threadPoolSize how many threads to use
     */
    public PerformanceTestResult runPerformanceTest(
            Runnable task,
            int executionCount, 
            int threadPoolSize) throws InterruptedException;
}

where PerformanceTestResult contains total time (how long the whole performance test took in total), minimum time (how long the shortest single execution took) and maximum time (how long the longest single execution took).

So, I learned many new things today - about thread pools, types Executors, ExecutorService, Future, CompletionService etc.

If I had Callable task, I could make next:

1. Return current time in the end of call() procedure.
2. Create some data structure (some Map may be) to store start time and Future object, that retuned by fixedThreadPool.submit(task) (do this executionCount times, in loop);
3. After execution I could just subtract start time from end time for every Future.

(Is this right way in case of Callable task?)

But! I have only Runnable task, so I continued looking. I even create FutureListener implements Callable<Long>, that have to return time, when Future.isDone(), but is seams little crazy for my (I have to double threads count).

So, eventually I noticed CompletionService type with interesting method take(), that Retrieves and removes the Future representing the next completed task, waiting if none are yet present., and very nice example of using ExecutorCompletionService. And there is my solution.

public class PerformanceTesterImpl implements PerformanceTester {


@Override
public PerformanceTestResult runPerformanceTest(Runnable task,
        int executionCount, int threadPoolSize) throws InterruptedException {
    long totalTime = 0;
    long[] times = new long[executionCount];

    ExecutorService pool = Executors.newFixedThreadPool(threadPoolSize);

    //create list of executionCount tasks 
    ArrayList<Runnable> solvers = new ArrayList<Runnable>();
    for (int i = 0; i < executionCount; i++) {
        solvers.add(task);
    }

    CompletionService<Long> ecs = new ExecutorCompletionService<Long>(pool);

    //submit tasks and save time of execution start
    for (Runnable s : solvers)
        ecs.submit(s, System.currentTimeMillis());

    //take Futures one by one in order of completing
    for (int i = 0; i < executionCount; ++i) {
        long r = 0;
        try {
            //this is saved time of execution start
            r = ecs.take().get();
        } catch (ExecutionException e) {
            e.printStackTrace();
            return null;
        }
        //put into array difference between current time and start time
        times[i] = System.currentTimeMillis() - r;
        //calculate sum in array
        totalTime += times[i];
    }

    pool.shutdown();
    //sort array to define min and max
    Arrays.sort(times);        

    PerformanceTestResult performanceTestResult = new PerformanceTestResult(
            totalTime, times[0], times[executionCount - 1]);
    return performanceTestResult;
}
}

So, what can you say? Thanks for replies.

Answer 1

I would use System.nanoTime() for higher resolution timings. You might want to ignroe the first 10,000 tests to ensure the JVM has warmed up.

I wouldn't bother creating a List of Runnable and add this to the Executor. I would instead just add them to the executor.

Using Runnable is not a problem as you get a Future<?> back.

Note: Timing how long the task spends in the queue can make a big difference to the timing. Instead of taking the time from when the task was created you can have the task time itself and return a Long for the time in nano-seconds. How the timing is done should reflect the use case you have in mind.

---

A simple way to convert a Runnable task into one which times itself.

finla Runnable run = ...
ecs.submit(new Callable<Long>() {
    public Long call() {
         long start = System.nanoTime();
         run.run();
         return System.nanoTime() - start;
    }
});

Answer 2

There are many intricacies when writing performance tests in the JVM. You probably aren't worried about them as this is an exercise, but if you are this question might have more information: How do I write a correct micro-benchmark in Java?

That said, there don't seem to be any glaring bugs in your code. You might want to ask this on the lower traffic code-review site if you want a full review of your code: http://codereview.stackexchange.com