4

I have updated my:

  • Ubuntu server to 16.04.1 LTS and
  • MONO to v4.6.2

...from official repository.

Since the update, the websites are still running fine, but after about a day or two, some of the MONO processes go crazy and take 100% of the CPU. I have different websites; mostly plain HTML with just a little bit of code. It happens randomly, and on different websites each time. It's totally random.

I then receive an email alert of high CPU usage, connect via SSH, type "htop", and kill the process and it's back to normal ... for a day or two.

This definitely looks like a bug in this version of MONO. Any way to fix it? Anyone else had this problem? Or perhaps I should switch to a different version that doesn't have this corruption?

Thanks

Edit: After 2 days, EVERY MONO process is taking up the full CPU.

Looking into the Apache2 log file, I could find this related to MONO

WARNING: WebConfigurationManager's LRUcache evictions count reached its max size
Cache Size: 100 (overridable via MONO_ASPNET_WEBCONFIG_CACHESIZE)

Also, "service apache2 restart" does not solve the problem. I must manually kill the processes, or reboot.

Etienne Charland
  • 3,424
  • 5
  • 28
  • 58
  • Not sure what you mean. I write using Visual Studio 2015 on Windows, and deploy on Ubuntu/MONO. The websites have been running fine for years, up until the updates. Mostly, I updated because I wanted to use SQLite which require v4.2.3.4 or higher. – Etienne Charland Feb 04 '17 at 16:19
  • Sorry it's just a joke. Have you tried another machine or different types of Mono apps? Otherwise it's generally bold to blame .net/mono for bugs. Wishing you well :) –  Feb 05 '17 at 01:48
  • Ohhhh...so is SQLite new to the equation? Maybe it is your new code for that? –  Feb 05 '17 at 01:49
  • That's what I thought at first ... until I realized the processes taking 100% CPU were *not* using SQLite at all. The issue isn't related to this new website. – Etienne Charland Feb 05 '17 at 03:22
  • Hmm, is it related to the Ubuntu update perhaps? Is it possible to try on the prior version? –  Feb 05 '17 at 03:45
  • There were 2 major updates: Ubuntu update and MONO update. Testing a prior version of Ubuntu would be near impossible. The best I can do is test various versions of MONO... once I have time to figure out how to do that. And testing a different machine? Setting up a new VPS is a pain! – Etienne Charland Feb 05 '17 at 04:01
  • _"until I realized the processes taking 100% CPU were not using SQLite at all"_ - were these programs written by you? Updated recently? Using the recent Mono? –  Feb 05 '17 at 04:17
  • Simple websites that have been running for years. Mostly a MasterPage and static content pages. No recent updates on these. There are websites with more server-side code but these are on other specific domains. – Etienne Charland Feb 05 '17 at 04:40
  • Is there a way to know what MONO is doing? A log file or something? This could tell which component is at fault. – Etienne Charland Feb 06 '17 at 13:49
  • I don't know sadly, I agree it would be useful. Is it possible to do remote debugging from VS on Windows to the app on Ubuntu? –  Feb 06 '17 at 14:34
  • 1
    Someone reported the same problem here http://stackoverflow.com/questions/30691619/mono-mod-mono-consuming-100-of-cpu – Etienne Charland Feb 06 '17 at 15:47
  • 2
    I also filed a bug here; no response yet https://bugzilla.xamarin.com/show_bug.cgi?id=52188 – Etienne Charland Feb 06 '17 at 15:48
  • 1
    'service apache2 reload' randomly causes MONO processes to run at 100%. First time it caused website A to go 100%. Second time was OK. Third time caused website B and C to go 100%. Furthermore, one of these websites is a test website that is never being called, and that hasn't even been JIT compiled. – Etienne Charland Feb 06 '17 at 15:59
  • As an update, MONO 2.8 still has the issue, but it appears to happen less often. I think I'll try switcher over to .NET Core. – Etienne Charland Mar 15 '17 at 01:58
  • Goodo. Keep us posted. Remember, if you find a solution, even if its an alternative, feel free to post as an answer :) –  Mar 15 '17 at 02:55
  • .NET Core now works on Ubuntu and should work for a basic ASP.NET website; just have to see what needs to change, and find the time to give it a try. – Etienne Charland Mar 15 '17 at 03:26
  • I was recommended *NOT* to install .NET Core as some have tried and it behaved in strange ways. My problem started happening when I both upgraded Ubuntu to 16.04.1 LTS and MONO to v4.6.2 (and v4.8). It's mostly related to the MONO version, but it's possible that the OS upgrade screwed something up too. MickyD, when did the problem start happening in your case? – Etienne Charland Mar 23 '17 at 19:07
  • I have not tried. I am just here to help if I can. :) –  Mar 24 '17 at 02:33

2 Answers2

1

After trying all options, it seems MONO just doesn't work well with Apache2 with mod_mono. The only solution I found is to switch Apache2 from prefork to worker mode, where the MONO server needs to be started manually and Apache2 simply forwards the requests to it -- and thus Apache2 doesn't directly touch MONO at all. There is very little documentation on how to do this, but since NGINX works in that mode, you can find instructions on how to set it up for NGINX and translate the app config file for Apache2.

These are good places to start

http://www.mono-project.com/docs/web/fastcgi/nginx/

http://epmjunkie.com/mono-fastcgi-startup-script/

Etienne Charland
  • 3,424
  • 5
  • 28
  • 58
0

I have played around with various MONO versions, and typing "service apache2 reload" to reproduce the high CPU usage problem.

In MONO 4.8, it seems to happen to happen a bit less often but the problem is still there.

In MONO 4.2.3.4, the problem is also there.

In MONO 4.2.1 that comes by default on Ubuntu, this problem doesn't happen.

As for .NET Core, some have tried it and highly recommended me to avoid it until it becomes more stable.

So for now, the only solution is to stick to MONO 4.2.1

This also confirms that this is related to MONO and not to my code or the server configuration.

Etienne Charland
  • 3,424
  • 5
  • 28
  • 58
  • Good investigation. You've spent a lot of time on this. Hope someone takes notice and makes a fix –  Mar 24 '17 at 02:36
  • This morning, my server was again at 100% CPU usage ... will keep you posted. There's definitely something different with this version though and it cannot be reproduced with a apache2 reload. If it keeps happening, I'll try with 3.2.8 – Etienne Charland Mar 24 '17 at 15:52
  • I gave up. This time I re-installed a fresh new VPN and transferred the files over. It might be the in-place OS upgrade that screwed things up; many recommend not to do that. This should solve it. – Etienne Charland Mar 27 '17 at 17:46
  • Shit. On a brand new Debian server, I still get the high CPU usage on 6 processes this morning. Except that now I'm not able to kill them. – Etienne Charland Mar 28 '17 at 15:35
  • The high cpu usage problem has been fixed in MONO 5.0 !! :D – Etienne Charland Jun 06 '17 at 13:59
  • @EtienneCharland it seems to be back in 5.16, I have two servers, one running 5.4 and one running 5.16, and the 5.16 will regularly crash and then reload, after which the mono process will go to 100% cpu and stay there until I kill it. – Anders Emil Nov 19 '18 at 10:21