2

Lots of personal experience, anecdotal evidence, and some rudimentary analysis suggests that a Java server (running, typically, Oracle's 1.6 JVM) has faster response times when it's under a decent amount of load (only up to a point, obviously).

I don't think this is purely hotspot, since response times slow down a bit again when the traffic dies down.

In a number of cases we can demonstrate this by averaging response times from server logs ... in some cases it's as high as 20% faster, on average, and with a smaller standard deviation.

Can anyone explain why this is so? Is it likely a genuine effect, or are the averages simply misleading? I've seen this for years now, through several jobs, and tend to state it as a fact, but have no explanation for why.

Thanks, Eric

Eric Bowman - abstracto -
  • 1,915
  • 1
  • 14
  • 24

1 Answers1

1

EDIT a fairly large edit for wording and adding more detail throughout.

A few thoughts:

  1. Hotspot kicks in when a piece of code is being executed significantly more than other pieces (it's the hot spot of the program). This makes that piece of code significantly faster (for the normal path) from that point forward. The rate of call after the hotspot compilation is not important, so I don't think this is causing the effect you are mentioning.

  2. Is the effect real? It's very easy to trick yourself with statistics. Not saying you are, but be sure that all your runs are included in the result, and that all other effects (such as other programs, activity, and your monitoring program are the same in all cases. I have more than one had my monitoring program, such as top, cause a difference in behaviour). On one occasion, the performance of the application went up appreciably when the caches warmed up on the database - there was memory pressure from other applications on the same DB instance.

  3. The Operating System and/or CPU may well be involved. The OS and CPU both actively and passively do things to improve the responsiveness of the main program as it moves from being mainly running to being mainly waiting for I/O and vice versa, including:

    1. OS paging memory to disk while it's not being used, and back to RAM when the program is running
    2. OS will cache frequently used disk blocks, which again may improve the application performance
    3. CPU instruction and memory caches fill with the active program's instruction and data

Java applications particularly sensitive to memory paging effects because:

  1. A typical Java application server will pre-allocate almost all free memory to Java. The large memory makes the application inherently more sensitive to memory effects
  2. The generational garbage collector used to manage Java memory ends up creating new objects over a lot of pages, so each request to the application will need more page requests than in other languages. (this is true principally for 'new' objects that have not been through many garbage collections. Objects promoted to the permanent generation are actually very compactly stored)
  3. As most available physical memory is allocated on the system, there is always a pressure on memory, and the largest, least recently run application is a perfect candidate to be pages out.

With these considerations, there is much more probability that there will be page misses and therefore a performance hit than environments with smaller memory requirements. These will be particularly manifest after Java has been idle for some time.

If you use Solaris or Mac, the excellent dTrace can trace memory and disk paging specific to an application. The JVM has numerous dTrace hooks that can be used as triggers to start and stop page monitoring.

On Solaris, you can use large memory pages (even over 1GB in size) and pin them to RAM so they will never be paged out. This should eliminate the memory page problem stated above. Remember to leave a good chunk of free memory for disk caching and for other system/maintenance/backup/management apps. I am sure that other OSes support similar features.

TL/DR: The currently running program in modern operating systems will appear to run faster after a few seconds as the OS brings the program and data pages back from disk, places frequently used disk pages in disk cache and the OS instruction and data caches will tend to be "warmer" for the main program. This effect is not unique to the JVM but is more visible due to the memory requirements of typical Java applications and the garbage collection memory model.

Andrew Alcock
  • 19,401
  • 4
  • 42
  • 60
  • Upvoting, but not convinced this answers the question. Lots of useful tips, though, thanks. I think what none of this completely answers (and which maybe my question doesn't make clear) is that a given service will run fine, then get some load and response times decrease, then when the load eases up, it slows down a little bit again. This is happens again and again and again, the same JVM, and without much else going on on the box. – Eric Bowman - abstracto - Dec 04 '12 at 10:53
  • Thanks for the upvote and clarification. Could you give some additional information: physical or virtual server? CPUs, CPU architecture and OS? How much RAM? App server used? Clustering? DB being used? Other services on the same system? Is the service *always* under at least some load? What are the observed performance figures? etc. This will give me a better idea of what is going on. – Andrew Alcock Dec 05 '12 at 07:53
  • Physical servers, 8 or 16 core CPUs. Actually I've seen this effect on a number of systems over the years, some clustered, some not, some with db, some with not. Although I have a very specific example of a relatively simple service in mind, my perception is this is a general characteristic of JVMs, and in mentioning it to colleagues, I'm not the only one who noticed, though none of us ever asked "why?" :) – Eric Bowman - abstracto - Dec 06 '12 at 00:05
  • Thanks for the description of the architecture(s) where you have observed this. I don't think I have personally seen this effect (other than for the paging as I described above). In the OP, you mentioned that you have figures that demonstrate this effect. Could you post them (redacted as necessary) and give other important details to enable a full interpretation? – Andrew Alcock Dec 06 '12 at 05:53