0

I am having an issue with my web application regularly crashing and resetting the application pool in IIS causing big performance issues, as well as wiping any timing threads running in my application.

The site is a .NET 4.5.2 C# MVC5 Site running on a 2012 Windows Server EC2 instance in AWS.

The issue was first noticed when I started see the site struggle to load after so many minutes of run-time. I thought it might be the ApplicationPool recycling and made sure to set IdleTime and Application Preload in IIS properly. The issue still persisted.

Next I went to the Server Manager to check the event logs and found these entries happening about every 15 minutes or so:

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96 Faulting module name: WMNetMgr.dll_unloaded, version: 12.0.9600.17415, time stamp: 0x545047db Exception code: 0xc0000005 Fault offset: 0x00000000000cf5cf Faulting process id: 0x17d0 Faulting application start time: 0x01d331dc20f096d0 Faulting application path: c:\windows\system32\inetsrv\w3wp.exe Faulting module path: WMNetMgr.dll Report Id: 777a35de-9dd1-11e7-81d7-025ff0be916d Faulting package full name: Faulting package-relative application ID:

and

WIN-7PCRJOFR05F 5011 Warning Microsoft-Windows-WAS System 9/20/2017 7:01:04 AM - A process serving application pool 'SiteName' suffered a fatal communication error with the Windows Process Activation Service. The process id was '6096'. The data field contains the error number.

Next I ran a DebugDiag2 Collection and Analysis:

WARNING - DebugDiag was not able to locate debug symbols for WMNetMgr.dll>, so the information below may be incomplete. In w3wp__SiteName__PID__5088__Date__09_20_2017__Time_06_31_02AM__436__Second_Chance_Exception_C0000005.dmp an access violation exception (0xC0000005) occured on thread 26 when another Module attempted to call the following unloaded Module: WMNetMgr.dll>.

Thread 26: Call Stack Unloaded_WMNetMgr.dll+cf5cf 0x000000de575cf7c0 0x000000dc2ed5ec10

This is the only error reported by this debugger. With no others exceptions in the .NET stack trace on the report. I can't seem to get the debugging symbols for this particular .dll and the messages do not seem to be very helpful.

The application utilizes WMPLib to create a singleton instance at startup of the wmplayer to play sounds on the Windows Server 2012 instance via web requests from clients. The application works in this regard with no issue playing sounds and requests from multiple users.

Here is the Singleton:

public sealed class SoundboardSingleton : IDisposable
{
    private static readonly Lazy<SoundboardSingleton> lazy =
        new Lazy<SoundboardSingleton>(() => new SoundboardSingleton());

    public static SoundboardSingleton Instance { get { return lazy.Value; } }

    public WindowsMediaPlayer WMPlayer;
    private StayAliveBot _liveBot;
    private Thread _botThread;

    private SoundboardSingleton()
    {
        WMPlayer = new WindowsMediaPlayer();
        WMPlayer.settings.volume = 50;

        _liveBot = new StayAliveBot();
        _botThread = new Thread(_liveBot.Live);
        _botThread.Start();
    }

    public void Dispose()
    {
        if (_botThread.IsAlive)
        {
            _botThread.Abort();
        }
    }
}

public class StayAliveBot
{
    public void Live()
    {
        while (SoundboardSingleton.Instance != null)
        {
            Thread.Sleep(1500000);
            SoundboardHelper.PlaySound("C:\\SoundboardOpFiles\\TestTone.wav");
        }
    }
}

and initially instantiated in Startup.cs via:

public partial class Startup
{
    public void Configuration(IAppBuilder app)
    {
        ConfigureAuth(app);

        // startup soundboard singleton
        SoundboardSingleton.Instance.WMPlayer.settings.volume = 50;
    }
}

I can run this application on my local dev machine with no issues or crashes. Everything functions as expected with no crashing. On deployment to the EC2 instance, everything on the site works properly, but there is now a crash / reset every 15 minutes.

My suspicion is either:

A) It is a problem with the WMPLib instance and some missing dependency on the Windows Server 2012 box that allows it to play sounds but causes crashes on regular intervals.

B) I've made a mistake with my singleton instantiation and it is somehow crashing my application.

I have tried the solution here but no results.

Any help would be appreciated.

edit: I have confirmed the issue is related to the usage of WMPLib as removing its use stopped the crashing every 15 minutes. Still not sure why this happens.

BagoDev
  • 313
  • 6
  • 12
  • Windows Media Player and its components are designed for desktop users only and in their user sessions only. Thus, when you run in a web app on IIS (in system session), anything can happen and is not supported by Microsoft. You have to use a third party library in this case, as I don't think Microsoft has any API specific to your case. – Lex Li Sep 21 '17 at 00:10
  • Technically this is for a desktop user, the audio is served to the Windows Server user and that sound is then routed via an interface to a hub where users are listening. – BagoDev Sep 21 '17 at 15:24
  • "Technically"? I am not sure how much you know the sessions and the underlying mechanism of Windows audio. Like I said, the related Windows API is not for ASP.NET. No more comments. – Lex Li Sep 21 '17 at 15:30
  • I only meant to clarify I was not serving this audio via web and it was playing on the Windows Server via Desktop Experience feature where it was captured by another application. I will try exploring something like https://github.com/naudio/NAudio – BagoDev Sep 21 '17 at 18:32

2 Answers2

2

This is not a direct answer to your question but rather a different way do to the same thing. Instead of the WMPLib COM control, try using the thread-safe MediaPlayer class from WPF. Add references to WindowsBase and PresentationCore, and use something like this instead:

using System.Windows.Media;

public void PlaySound(string filename)
{
    var mplayer = new MediaPlayer();
    mplayer.MediaEnded += new EventHandler(MediaEndedHandler);
    mplayer.Open(new Uri(filename));
    mplayer.Play();
}

public void MediaEndedHandler(object sender, EventArgs e)
{
    ((MediaPlayer)sender).Close();
}

You can also use it as a singleton the same way as above, and it is fully thread-safe, which WMPLib is not.

Documentation here.

Edit:

As noted in the comments, you really could use just a static class with a public bool property to show busy signal. A static class in IIS is shared among all requests for an application and the class is only subject to garbage collection when the application pool is recycled, so you do need to be careful with the lifetime of the objects you store in it, to avoid memory consumption problems. This code will use a new instance of the media player class for each PlaySound() and disposes of it right after it's done playing, but the busy flag is common among all requests made to the server.

using System;
using System.Threading;
using System.Windows.Media;

namespace SoundBoardApp
{
    public static class Soundboard
    {
        private static bool _isBusy = false;

        public static bool IsBusy { get { return _isBusy; } }

        private static void MediaEndedHandler(object sender, EventArgs e)
        {
            _isBusy = false;
            var wmp = ((MediaPlayer)sender);
            wmp.MediaEnded -= new EventHandler(MediaEndedHandler);
            wmp.Close();            
        }

        public static bool PlaySound(string filename)
        {
            if (!_isBusy)
            {
                _isBusy = true;
                var wmp = new MediaPlayer();
                wmp.MediaEnded += new EventHandler(MediaEndedHandler);
                wmp.Volume = 0.5;
                wmp.Open(new Uri(filename));
                wmp.Play();
                return true;
            }
            else
            {
                return false;
            }            
        }

    }

    public class StayAliveBot
    {
        public void Live()
        {
            while (true)
            {
                Thread.Sleep(1500000);
                if (!Soundboard.IsBusy) Soundboard.PlaySound("C:\\SoundboardOpFiles\\TestTone.wav");
            }
        }
    }
}
Drunken Code Monkey
  • 1,796
  • 1
  • 14
  • 18
  • Thanks for the response, I will give it a shot. Any idea if this will have issues on a Windows Server 2012 instance? – BagoDev Sep 20 '17 at 23:59
  • There is no reason I can see that it would. As long as IIS is configured with a managed thread pool and Windows Media Player is functional, it should work. – Drunken Code Monkey Sep 21 '17 at 00:05
  • The namespace is for WPF only, https://msdn.microsoft.com/en-us/library/system.windows.media(v=vs.110).aspx Using such in a web app would lead to unknown results, and is usually not supported by Microsoft at all. – Lex Li Sep 21 '17 at 00:09
  • It works fine, I have used it plenty of times before. Sure, it's in a WPF library, but that's where the story ends. This is just a thin managed interface to the Windows Media Player. Besides, this is the only other choice other than resorting to the antiquated and finnicky WMPLib COM interface, unless you go with external components. – Drunken Code Monkey Sep 21 '17 at 00:11
  • Working on adapting the singleton to this now. Reason I used singleton for this was for checking states on the player so I could send feedback to users trying to play a sound when another one is playing. Getting denial on the accessing the object because of thread ownership, and Invoke() is not being nice to me. – BagoDev Sep 21 '17 at 03:35
  • You shouldn't really expose the player object to the outside, instead create properties on your soundboard singleton. You could also just put everything in a simple static class and forego the singleton altogether. I have added an example above. – Drunken Code Monkey Sep 21 '17 at 04:24
  • Thank you again for providing a solution to this problem, I didn't know statics would work that way across all users. Only issue now seems to be the Event for MediaEnded is not firing. This question seems to reflect that same problem https://stackoverflow.com/questions/38436779/c-sharp-mediaplayer-mediaended-event-not-firing , there is a consensus for the MediaElement needing to be added to the VisualTree, but as this is without UI, and we are not using a MediaElement, I do not think it applies here. I'll start working the problem again in the morning. – BagoDev Sep 21 '17 at 07:36
  • I've tried this and NAudio, both static and singleton design, both are not having their EventHandlers fire for play back stopping. Still investigating. – BagoDev Sep 21 '17 at 18:48
  • I think the issue is that an ASP.NET thread does not have a dispatcher attached to it, while both WinForms and WPF support the Dispatcher. Set a breakpoint in your code after the creation of the MediaPlayer, and see what the Dispatcher property references. If it is null, then that is why your event is not firing. Try to instantiate the MediaPlayer inside a Thread or a Task, and see if that works any better. – Drunken Code Monkey Sep 21 '17 at 19:41
  • I managed to get it working using the answer I posted, thank you for the help. – BagoDev Sep 21 '17 at 22:22
1

I ended up using NAudio with my singleton pattern.

As per recommendations from Lex Li, I used a third party as the Windows.MediaPlayer was not meant for a web app. Based of the solution from Drunken Code Monkey, I used a boolean flag on the singleton to evaluate play states being checked frequently by a separate thread evaluating the PlaybackState.Stopped value on the IWavePlayer object in my singleton. My only concern is performance. I haven't noticed any problems yet, but I'm pretty sure managing Events in a Handler would have been more performant and a lot less code if it's even possible to do from a web app.

Here is the code:

using NAudio.Wave;

public sealed class SoundboardSingleton : IDisposable
{
    private static readonly Lazy<SoundboardSingleton> lazy =
        new Lazy<SoundboardSingleton>(() => new SoundboardSingleton());

    public static SoundboardSingleton Instance { get { return lazy.Value; } }

    public IWavePlayer WaveOutDevice { get; set; }
    public AudioFileReader AudioFileReaderObj { get; set; }
    public float Volume { get; set; }

    private MediaCloser _mediaCloser;
    private Thread _mediaCloserThread;
    private StayAliveBot _liveBot;
    private Thread _botThread;

    public bool IsBusy { get; set; }

    private SoundboardSingleton()
    {
        // checks our NAudio WaveOutDevice playback for stop
        _mediaCloser = new MediaCloser();
        _mediaCloserThread = new Thread(_mediaCloser.CheckForStoppedPlayback);
        _mediaCloserThread.Start();

        // thread to play sound every 25 minutes, to avoid idle flag
        _liveBot = new StayAliveBot();
        _botThread = new Thread(_liveBot.Live);
        _botThread.Start();
    }

    public bool PlaySound(string filename)
    {
        // make sure we are not active
        if (IsBusy) { return false; }

        // process sound
        IsBusy = true;
        WaveOutDevice = new WaveOutEvent();
        AudioFileReaderObj = new AudioFileReader(filename);
        AudioFileReaderObj.Volume = Volume;
        WaveOutDevice.Init(AudioFileReaderObj);
        WaveOutDevice.Play();

        return true;
    }

    public void CloseWaveOut()
    {
        // clean up sound objects
        WaveOutDevice?.Stop();

        if (AudioFileReaderObj != null)
        {
            AudioFileReaderObj.Dispose();
            AudioFileReaderObj = null;
        }
        if (WaveOutDevice != null)
        {
            WaveOutDevice.Dispose();
            WaveOutDevice = null;
        }
    }

    public void Dispose()
    {
        if (_mediaCloserThread.IsAlive)
        {
            _mediaCloserThread.Abort();
        }
        if (_botThread.IsAlive)
        {
            _botThread.Abort();
        }
    }
}

public class MediaCloser
{
    public void CheckForStoppedPlayback()
    {
        while (true)
        {
            // continuously check for our stopped playback state to cleanup
            Thread.Sleep(500);
            if (SoundboardSingleton.Instance.WaveOutDevice != null &&
                SoundboardSingleton.Instance.WaveOutDevice.PlaybackState == PlaybackState.Stopped)
            {
                SoundboardSingleton.Instance.CloseWaveOut();
                SoundboardSingleton.Instance.IsBusy = false;
            }
        }
    }
}

public class StayAliveBot
{
    public void Live()
    {
        while (true)
        {
            // prevent bot from going idle
            Thread.Sleep(1500000);
            if (!SoundboardSingleton.Instance.IsBusy)
            {
                SoundboardSingleton.Instance.PlaySound(ConfigurationManager.AppSettings["SoundboardHeartbeatFile"]);
            }
        }
    }
}

Hopefully this helps anyone running into the same problems. My site has been up and running for a few hours now with no issues and clients spamming the board. Thanks again everyone who helped.

BagoDev
  • 313
  • 6
  • 12