C# server scalability issue on linux

Question

I've a C# server developed on both Visual Studio 2010 and Mono Develop 2.8. NET Framework 4.0

It looks like this server behaves much better (in terms of scalability) on Windows than on Linux. I tested the server scalability on native Windows(12 physical cores), and 8 and 12 cores Windows and Ubuntu Virtual Machines using Apache's ab tool.

The windows response time is pretty much flat. It starts picking up when the concurrency level approaches/overcomes the number of cores.

For some reason the linux response times are much worse. They grow pretty much linearly starting from level 5 of concurrency. Also 8 and 12 cores Linux VM behave similarly.

So my question is: why does it perform worse on linux? (and How can I fix that?).

Please take a look at the graph attached, it shows the averaged time to fulfill 75% of the requests as a function of the requests concurrency(the range bar are set at 50% and 100%).

I have a feeling that this might be due to mono's Garbage Collector. I tried playing around with the GC settings but I had no success. Any suggestion?

Some additional background information: the server is based on an HTTP listener that quickly parses the requests and queues them on a thread pool. The thread pool takes care of replying to those requests with some intensive math (computing an answer in ~10secs).

What is a "C# server"? Is that different from a server? I mean, in your title, is "C#" being used as an adjective? Is a "C# server" a kind of server? — John Saunders, May 19 '12 at 00:58
>some intensive math (computing an answer in ~10secs). I think your problem is there.. It's not typical scenario for a server. How many cores your physical hardware have, 12? How the graph looks after the 10 concurrent requests? — Soonts, May 21 '12 at 02:22
As I know, mono isn't suited for these uses. The development of mono is focused in small desktop applications. I get this information from a developer of mono, I will try to find more information and will post here if I find. — sergiogarciadev, May 28 '12 at 17:18
Apart from the sgen garbage collector you should also try running the newest (i.e. trunk, compiled from repository) version of Mono, there were a lot of improvements lastly. — konrad.kruczynski, May 31 '12 at 08:26

Mitch Wheat · Answer 1 · 2012-05-19T01:08:56.097

4

You need to isolate where the problem is first. Start by monitoring your memory usage with HeapShot. If it's not memory, then profile your code to pinpoint the time consuming methods.

This page, Performance Tips: Writing better performing .NET and Mono applications, contains some useful information including using the mono profiler.

Excessive String manipulation and Boxing are often 'hidden' culprits of code that doesn't scale well.

edited May 19 '12 at 01:08

answered May 18 '12 at 23:47

Mitch Wheat

295,962
43
465
541

Thanks Mitch.I'm gonna checkout the tools/docs you suggested. But I can tell you that the server was already (intensively) profiled and optimized on windows(using eqatec profiler). The server behavior on Windows is great:flat response(tested with up to 12 concurrent requests on a 12 physical cores machine). Instead it performs much worse on linux.A memory bandwidth bottleneck should affect linux and windows machine the same way, this is why I ruled it out. Please take a look at the graph I've uploa. I couldn't attach it before because of stackoverflow anti-spam policies (I'm a new user) – Enio May 19 '12 at 15:43
I used the tool you suggested (HeapShot) to profile the GC behavior. It looks like there are ~ 10 garbage collections for every 20 requests I send to the server. Each one of these GC events handles less than 10Mb of RAM. Most of the work is done freeing System.Single[]. anyway it looks to me that these memory events are too rare to impact the server scalability. What do you think? – Enio May 22 '12 at 00:30

score 1 · Answer 2 · answered May 21 '12 at 02:13

1

Try the sgen garbage collector (and for that, Mono 2.11.x is recommended). Look at the mono man page for more details.

answered May 21 '12 at 02:13

knocte

16,941
11
79
125

Thanks. I tried running with mono --gc=sgen . Unfortunately the behavior is still the same – Enio May 22 '12 at 00:07
another option you could try is the LLVM backend – knocte May 22 '12 at 23:34

score 1 · Answer 3 · answered May 21 '12 at 02:27

1

I don't believe it's because of the GC. AFAIK the GC side effects should be more or less evenly distributed across the threads.

My blind guess is: you can fix it by playing with ThreadPool.SetMinThreads/SetMaxThreads API.

answered May 21 '12 at 02:27

Soonts

20,079
9
57
130

I tried varying the number of min/max working/io threads and it seems not to have a (positive) effect on the scalability. But thanks anyway. Btw the effect is evenly distributed across all threads: please take a look at the error bars on the brown and orange curves (the ones on mono) at concurrency 9-10. you can see how the reply times are pretty much the same for all requests. – Enio May 22 '12 at 00:20
OK, my blind guess #2: the I/O completion ports (used by .NET CLR thread pool) were invented in 1993 (Windows NT 3.5) and since that time are fine tuned & debugged. OTOH, the epoll (IOCP clone, used by MONO for same purpose) is only here since 2002, so the Windows IOCP is just better. On Linux, everything's good until concurrent requests * 2 < CPU cores. Looks like the thread who parsed the request is still busy doing something, not ready to grab a work item from the thread pool.. – Soonts May 30 '12 at 00:40
BTW, when benchmarking, do you start all requests simultaneously? What changes if you'll launch them in 1-2 seconds intervals one after another? – Soonts May 30 '12 at 00:41

score 1 · Answer 4 · answered May 21 '12 at 02:51

I would strongly recommend you do some profiles on the code with regards to how long individual methods are running. It's quite likely that you're seeing some locking or similar multi-threading difficulties that are not being handled perfectly by mono. CPU and RAM usage would also help.

score 1 · Answer 5 · edited May 23 '17 at 12:01

1

I believe this may be the same problem that we tracked down involving the thread pool and the starting behavior for new threads, as well as a bug in the mono implementation of setMinThreads. Please see my answer on that thread for more information: https://stackoverflow.com/a/12371795/1663096

edited May 23 '17 at 12:01

Community

1
1

answered Oct 29 '12 at 04:11

Jake Moshenko

273
3
8

score 0 · Answer 6 · answered May 21 '12 at 02:31

0

If you code throws a lot of exceptions, then mono is 10x faster than .NET

answered May 21 '12 at 02:31

linquize

19,828
10
59
83

2

that's awesome! But my issue is the opposite: the server scalability is much better on windows than on mono/linux. – Enio May 22 '12 at 00:09

C# server scalability issue on linux

6 Answers6