7

I'm trying to figure out why our software runs so much slower when run under virtualization. Most of the stats I've seen, say it should be only a 10% performance penalty in the worst case, but on a Windows virtual server, the performance penalty can is 100-400%. I've been trying to profile the differences, but the profile results don't make a lot of sense to me. Here's what I see when I profile on my Vista 32-bit box with no virtualization: enter image description here

And here's one run on a Windows 2008 64-bit server with virtualization:enter image description here

The slow one is spending a very large amount of it's time in RtlInitializeExceptionChain which shows as 0.0s on the fast one. Any idea what that does? Also, when I attach to the process my machine, there is only a single thread, PulseEvent however when I connect on the server, there are two threads, GetDurationFormatEx and RtlInitializeExceptionChain. As far as I know, the code as we've written in uses only a single thread. Also, for what it's worth this is a console only application written in pure C with no UI at all.

Can anybody shed any light on any of this for me? Even just information on what some of these ntdll and kernel32 calls are doing? I'm also unsure how much of the differences are 64/32-bit related and how many are virtual/not-virtual related. Unfortunately, I don't have easy access to other configurations to determine the difference.

nmichaels
  • 49,466
  • 12
  • 107
  • 135
Morinar
  • 3,460
  • 8
  • 40
  • 58
  • 6
    It seems like a good starting point would be to run the same OS in your virtualized environment as in your non-virtualized environment. Then you'd at least be comparing apples to pears, instead of to horses. – nmichaels Apr 15 '11 at 20:33
  • You need to add more info, that's far too deep into kernel space to get an idea of what you're doing. – Sean Apr 15 '11 at 20:34
  • 1
    Sounds http://serverfault.com/ to me. – Teson Apr 15 '11 at 20:35
  • It seems pretty hard to get information on this `RtlInitializeExceptionChain`... – Alexandre C. Apr 15 '11 at 20:37
  • Those are the exact 2 comments I was most worried I'd get first. @nmichaels - I'm trying to get an apples to apples comparison in place, but it's slow going and was hoping to get some information before I had it. @Sean - In short, this app does nothing more than print out every record in a CTree database table. – Morinar Apr 15 '11 at 20:37
  • 3
    This doesn't sound like serverfault to me -- it's clearly about what they've done in their software that makes it slower, and how to modify it to avoid that. IOW, it's about writing the software to run well in a couple of specific environments. – Jerry Coffin Apr 15 '11 at 20:42
  • @Morinar - How big is the database? It could be an IO bottleneck. Virtualization does have an impact on IO performance, which would explain the ``WaitForMultipleObjects`` and ``WriteFile`` – Sean Apr 15 '11 at 20:45
  • @Sean - It's a small number of MB in size. The table I'm benchmarking on has 14k records. – Morinar Apr 15 '11 at 20:53

2 Answers2

6

I suppose we could divide reasons for slower performance on a virtual machine into two classes:

1. Configuration Skew

This category is for all the things that have nothing to do with virtualization per se but where the configured virtual machine is not as good as the real one. A really easy thing to do is to give the virtual machine just one CPU core and then compare it to an application running on a 2-CPU 8-core 16-hyperthread Intel Core i7 monster. In your case, at a minimum you did not run the same OS. Most likely there is other skew as well.

2. Bad Virtualization Fit

Things like databases that do a lot of locking do not virtualize well and so the typical overhead may not apply to the test case. It's not your exact case, but I've been told the penalty is 30-40% for MySQL. I notice an entry point called ...semaphore in your list. That's a sign of something that will virtualize slowly.

The basic problem is that constructs that can't be executed natively in user mode will require traps (slow, all by themselves) and then further overhead in hypervisor emulation code.

Community
  • 1
  • 1
DigitalRoss
  • 143,651
  • 25
  • 248
  • 329
  • Those are some greats tips. The second one in particular seems startlingly likely. Curious to hear any thoughts anybody else might have. – Morinar Apr 15 '11 at 20:53
0

I'm assuming that you're providing enough resources for your virtual machines, the benefit of virtualization is consolidating 5 machines that only run at 10-15% CPU/memory onto a single machine that will run at 50-75% CPU/memory and which still leaves you 25-50% overhead for those "bursty" times.

Personal anecdote: 20 machines were virtualized but each was using as much CPU as it could. This caused problems when a single machine was trying to use more power than a single core could provide. Therefore the hypervisor was virtualizing a single core over multiple cores, killing performance. Once we throttled the CPU usage of each VM to the maximum available from any single core, performance skyrocketed.

saschabeaumont
  • 22,080
  • 4
  • 63
  • 85