3

I have a set of 196 test methods. The Execution time of these testcases vary every time I run it. It has been run in a controlled environment,Say,For garbage collection, I included null in teardown().

Every time before executing the tests, I also make sure CPU usage, Memory usage, Disk space, System load are same for every start.

Also,The time variation is not in any particular order. I need to know why don't we get stable execution time while executing the same test cases again?

I made 93 cases stable by including a warm up period in the class. Other cases are related to database connections (reading a data or updating a data in database). Is it possible to have same execution time every time i run these testcases. (Execution time refers to junit testcase execution time)

tps
  • 51
  • 5
  • How much do they vary by, is it a matter of milliseconds or seconds? – JensB Jun 02 '15 at 08:14
  • What happens in these tests? How do you run these tests? Are they always executed in the same order? – reto Jun 02 '15 at 08:14
  • 2
    There is many more parameters and influences. E.g. CPU caches, OS caches,... Better to repeat test many times and measure total time. – JiriS Jun 02 '15 at 08:16
  • Depending on the cases.. For some small tests it varies around 10ms. For some tests like connection of Application servers it varies by 40seconds. – tps Jun 02 '15 at 08:16
  • 3
    Mandatory link for this topic: http://stackoverflow.com/questions/504103/how-do-i-write-a-correct-micro-benchmark-in-java – reto Jun 02 '15 at 08:20
  • 2
    40 seconds out of how many? Also, when you say "connection of application servers" --- is that hitting the network? – Patrick Collins Jun 02 '15 at 08:32
  • Yes it hits the network. Also I repeated the test so many times and found the difference between each two time execution. @JiriS I ll check with CPU caches, OS caches now. – tps Jun 02 '15 at 08:49
  • 1
    If it hits the network is the load of application servers constant as well? Do you use only local network or is it communicating through the Internet? Even if it's using local network only it might be still highly influenced by the load of the remote server/load of the network/load of switches/routers/... – JiriS Jun 02 '15 at 08:52
  • Out of 196, 5 testcases have same execution time ( the Execution time of these testcases lies between 1 to 3 ms), 25 testcases have a time difference between 2 to 10 ms,remaining 106 cases varies much. three cases have variation around 60 seconds. This cases involve sql queries to connect to a db and updating the data in it. – tps Jun 02 '15 at 09:00
  • 2
    @tps You shouldn't write everything in the comments. It belongs to your question, just edit it. – maaartinus Jun 02 '15 at 12:34

1 Answers1

1

Two primary things come to mind with Java performance:

  • You need to warmup the JVM, your tests are being interpreted as bytecode and are at the mercy of the JVM. That means executing the same test upwards of thousands of times during the same run.
  • JUnit tests are not measured with much accuracy. In fact, it's pretty much impossible to get an exact performance reading, even with libraries build specifically for this. This is why taking an average of multiple samples is generally suggested.

Yet, this and those others suggested by Reto are just what could be causing variance by Java. Where a variance of milliseconds is more than expected. For an example of this, create a unit test that takes a thread and puts it to sleep for 10 ms. Watch as you're given results anywhere from 7 ms to 13 ms to 17 ms or more. It just isn't a reliable way to measure things.

If you're connecting to a network, uploading data to a database, etc. I can't speak on behalf of that, but you need to take into account the variance of those systems as well.

I would suggest breaking your three tests with the greatest variance into smaller blocks. Try and isolate where your biggest bottleneck is, then concentrate on optimizing that operation or set of operations. I would think that connecting to the database takes the greatest amount of time, next to that would likely be executing the query. But you should isolate the measurement of these operations to make sure of that.

Community
  • 1
  • 1
wateryan
  • 40
  • 9