8

We are referencing a 3rd party proprietary CLI DLL in our .net project. This DLL is only an interface to their proprietary C++ library. Our project is an asp.net (MVC4/Web API) web application.

The C++ unmanaged library is rather unstable. Sometimes it crashes with e.g. dangling pointers. We have no way of solving it, and using this library is a first-class customer requirement.

When the application crashes, the application pool in IIS doesn't respond anymore. We have to restart it, and doing so takes a couple minutes (yes, that long!).

We would like to keep this unstable DLL from crashing our application. What's the best way of doing it? Can we keep the CLI DLL in a separate AppDomain? How?

Thanks in advance.

Schiavini
  • 2,869
  • 2
  • 23
  • 49
  • What have you observed? Does the process hang after a certain number of calls? Do you suspect the unstable DLL has memory leaks? – xpereta Jun 10 '13 at 09:23
  • The memory leaks haven't been a problem until now. During development, the DLL sometimes crashes the development server with a memory access violation. Sometimes the development server uses ~70% of the CPU until I kill the process. When deployed in IIS, the crashing and CPU problems are not really happening that much, but once in a while the IIS process stops responding, and a IIS restart takes a long time. – Schiavini Jun 11 '13 at 07:34
  • Have you tried increasing the IIS process recycling rate? – xpereta Jun 13 '13 at 08:50
  • That wouldn't help since the problem is that we get dangling pointers - recycling would help with memory leaks – Schiavini Jun 13 '13 at 09:34

4 Answers4

6

I think every answer to this question will be some kind of work around.

My workaround would be to not interact directly with the DLL from your web application.

Instead write your requests from the web application to either a Message Queue or a SQL table. You can then have another application such as a Windows Service which reads the requests, interacts with the DLL and then writes the results back for your web application to read.

I'm not saying that SQL / Message Queues are the right way, I'm more thinking of the general process flow.

goughy000
  • 668
  • 4
  • 13
4

I had this exact problem with a third party library that accessed protected memory for purposes of interacting with a hardware copy protection dongle. It worked fine in a console or winforms app, but crashed like crazy when called from an IIS application.

We tried several different things, some of which are mentioned in other answers on this page. But ultimately, the best solution for us was to us a very old technology - .Net Remoting. I know - it's somewhat frowned on these days. But it fit this particular need quite well.

The unstable code was placed in a Windows Service application. The web application made remoting calls to this service, which relayed the commands to the third-party library.

Now I'm sure you could do the same thing with WCF, sockets, etc. But remoting was quick and easy to setup, and since we only talk to the same server it works without opening any ports. It just talks on a named pipe.

It does mean a second service to install besides the web application, but that was acceptable in my particular use case.

If you did something similar, and the third-party code actually crashed the service, you could probably write some code in your main application to bring it back up.

So perhaps a process boundary is more useful than an App Domain when you have unstable code to wrangle.

Matt Johnson-Pint
  • 230,703
  • 74
  • 448
  • 575
2

I would first increase the IIS process recyling rate, maybe the the DLL code fails after a certain number of calls, or after the process reaches a certain amount of memory usage.

You can find information on the configuration of IIS 7.0 recycling options here: http://technet.microsoft.com/en-us/library/cc753179(v=ws.10).aspx

In your case I would recycle the process at a specific time, when you know there is less load on the application. And after a certain number of requests (lower than the default) to try and have "fresh" process most of the time.

The recycling process is graceful in the sense that the the old process is not terminated until the one that will replace it is ready, so there should be no noticeable downtime. More information about the recycling mechanism here: http://technet.microsoft.com/en-us/library/cc745955.aspx

If the above does not solve the problem I would wrap the calls in my own code that manages the unstable DLL execution.

This code should recover from the failures for example by repeating the failing calls until a result is obtained and failing with a graceful error if it is not possible after a number of attempts.

Internally the calls to the unstable DLL could be made in a spawned thread or even the code could be in an new external executable that you could launch with Process.Start.

This last option has more overhead but it might be your only option. See this SO question for more information on this: How do you handle a thread that has a hung call?

Community
  • 1
  • 1
xpereta
  • 692
  • 9
  • 21
2

I suggest following solution.

  1. Wrap this dll with another web application. Can be one of the following ones. Since you already use web api, it is most suitable for you.

    1. Simple ASMX Web Service
    2. WCF Service
    3. Asp.Net MVC - WEB Api Service

  2. Control your p-invoke code so that you do not have any bug? See following articles.

    1. The Black Art of P/Invoke and Marshaling in .NET
    2. P/Invoke Revisited
  3. Publish this application to IIS with different application pool.
  4. Use standard techniques suggested before like. I suggest configure recycling IIS for both memory and scheduled times.
    1. IIS process recycling rate
    2. How to limit the memory used by an application in IIS?
Community
  • 1
  • 1
Atilla Ozgur
  • 14,339
  • 3
  • 49
  • 69
  • I don't see how wrapping with another web application would help me. IIS would still freeze. – Schiavini Jun 17 '13 at 10:44
  • Every application pool is different process. You main application in a different pool, that is different process continues to work. – Atilla Ozgur Jun 17 '13 at 10:46
  • But it cannot recover from a crash from the other application since the process will be stuck – Schiavini Jun 17 '13 at 10:51
  • When you call other application and other process stuck, your web call will timeout. It will not stuck, it is same as calling an unresponsive or not working application. – Atilla Ozgur Jun 17 '13 at 12:37