5

I'm writing some c# that will load a third party assembly.

If the third party decided to be malicious, they could write a recursive function that would end up in a StackOverflowException, bringing down my application.

Is it possible to detect a recursive function?

Update: For undesirable sitations like while(true), or for(;;), I already have a solution. Essentially, I run the third party code in a separate thread, and if the thread takes longer than a fixed duration, I pull the plug. This doesn't work well with recursion since the stack limit is reached extremely quickly.

Update: Perhaps I've misrepresented the solution that I'm after. If I end up getting a lot of intentionally malicious code, I'll change the application to run the third party code in a separate process. However at this stage, I'm assuming that the code will only cause problems because it's poorly written.

Accepted Answer I've decided that the best approach would be to run the third party libraries in a separate process. I can have multiple instances of the processes running, and even do a sort of load balancing of my third party libraries across the processes. If malicious code is executed that kills one of the processes, I should be able to detect which library killed it, mark that library as malicious, and relaunch the process with all of the non-malicious libraries.

Thanks for everyone's great suggestions!

Robert
  • 1,487
  • 1
  • 14
  • 26
  • 1
    Do you also plan to guard against `for(;;);`? – cdhowie Dec 22 '10 at 04:16
  • 4
    @cdhowie Well, once you solve the halting problem, that's a piece of cake ;-) –  Dec 22 '10 at 04:17
  • `for(;;);` will just slow down the computer, not crash the app like a StackOverflow exception. – Gabe Dec 22 '10 at 04:18
  • @Gabe: True enough. What about something like `var foo = new List(); for(;;) foo.Add(new object());`? The point I'm trying to make is that there are a whole slew of potentially undesirable actions that might not be easily detectable without actually executing the code. – cdhowie Dec 22 '10 at 04:20
  • possible duplicate of [How do I prevent and/or handle a StackOverflowException? (C#)](http://stackoverflow.com/questions/206820/how-do-i-prevent-and-or-handle-a-stackoverflowexception-c) – Byron Whitlock Dec 22 '10 at 04:31
  • cdhowie: You can catch an OutOfMemory exception. You can't catch a StackOverflow. This is unfortunate because it makes it much harder to write a recursive-descent parser that works with deeply-nested inputs. – Gabe Dec 22 '10 at 04:31
  • @Byron: I think the OP's concerns (that the app is malicious) is rather different than the concern of the question you mention, even though the question itself is the same. Since the two questions cover very different ground, I don't think this is really a duplicate. – Brian Dec 22 '10 at 14:14
  • for undesirable sitations like while(true), or for(;;), I already have a solution. Essentially, I run the third party code in a separate thread, and if the thread takes longer than a fixed duration, I pull the plug. This doesn't work well with recursion since the stack limit is reached extremely quickly. – Robert Dec 22 '10 at 19:12

5 Answers5

8

It's not easy to do that in the general case. Besides, recursion is a useful tool for programming and it's not a good idea to ban that completely.

A better idea is to run the assembly in another process and use an interprocess communication mechanism to call methods from the trusted process.

Mehrdad Afshari
  • 414,610
  • 91
  • 852
  • 789
  • 4
    Or use a separate AppDomain. You can kill off the entire AppDomain if it starts misbehaving. – cdhowie Dec 22 '10 at 04:15
  • 2
    @cdhowie: running in separate `AppDomain`s is not entirely safe either. There are sneaky ways to take down the whole application. For one instance, see: http://stackoverflow.com/questions/3906282/can-i-prevent-an-uncaught-exception-in-another-appdomain-from-shutting-down-the-a In general, AppDomains isolate data pretty well, but they don't isolate threads, and that's the root of the problem. – Mehrdad Afshari Dec 22 '10 at 04:17
  • Running the library in a separate AppDomain is an option, but wouldn't that be a very heavy-weight solution? I expect to be able to run loads of instances of these third party libraries, so this might not work out when numbers get large.. – Robert Dec 22 '10 at 19:10
  • Well, Google Chrome is doing it – Mehrdad Afshari Dec 22 '10 at 22:34
  • @Mehrdad Yes it is! But I'm thinking of hundreds to thousands. Not ~10! – Robert Dec 22 '10 at 22:52
  • @Robert you can consolidate all of the addins in a single process. – Mehrdad Afshari Dec 22 '10 at 23:00
  • Well, if one rogue one brings them all down, then they might as well bring down the server. However, I like that idea. I could split them into "process pools" whereby each process can host a number of them. Thanks for the great idea! – Robert Dec 24 '10 at 08:05
8

Suppose you found a way to do the impossible and detect recursions. Great. Does that help? No. Nothing is stopping the hostile assembly from simply throwing the exception with a throw statement.

Besides, this is the least of your problems. If you have untrusted hostile code they are going to do a whole lot more nastiness than simply throwing an exception to take down the process. They're going to be trying to steal secret information, install rootkits, you name it. The last thing hostile code wants to do is throw an exception; doing so calls attention to the hostile code. It triggers automatic reports that will be analyzed. Authors of hostile code wants to avoid detection, not loudly call attention to the attack!

If you have a partially trusted third-party assembly then use the tools we have put at your disposal for that exact scenario. Rather than trying to solve impossible problems yourself, spend your valuable time using the Code Access Security system for what it was designed for: to handle partial-trust code.

Probably what you want to study up on is MEF, which is a framework designed by the VSTO team for handling managed add-ons that might have partial trust. (I did some of the early design and security review work for MEF many years ago but I left the team early on and am not an expert on it by any means.)

Eric Lippert
  • 647,829
  • 179
  • 1,238
  • 2,067
  • Even if I use the CAS system to prevent untrusted code from accessing the file system, using reflection, e.t.c, I still can't prevent the code from just locking up the application with a recursive function. – Robert Dec 22 '10 at 19:07
  • I have a general solution for preventing for (;;), and throwing exceptions. The code is run in a separate thread, and is wrapped in a try catch. Even if there are still ways to get around this, I'm trying to go for the 80% solution, rather than trying to handle all possible scenarios. I feel that recursive loops (malicious, or unintentional) fall into that 80%. – Robert Dec 22 '10 at 19:15
  • 2
    @Robert: going for 80% is a fundamentally bad attitude to have about security. Suppose your bank decided to repel only 80% of attacks against their vaults. **Motivated attackers will find the unprotected 20%**. If you want to keep an attacker from taking down your process then *do not run their code in your process*. Run it in its own appdomain, or even better, in its own process (or even better still, its own virtual machine.) Go for the 100% solution; you want to make it *impossible* for an attacker to harm your user. – Eric Lippert Dec 22 '10 at 19:20
  • @Eric I agree whole heartedly; perhaps I need to rephrase my question. I'm assuming that my third party software will not be intentionally malicious. If it is, I will deal with that on a case by case basis. If this becomes too much of an overhead, I'll look at running it in a separate process – Robert Dec 22 '10 at 19:27
  • 1
    If a "hostile" assembly just throws a `StackOverflowException` that's not a problem because you can catch it. It's only when the stack actually overflows that you're screwed. – Gabe Dec 22 '10 at 20:55
0

You could use reflection on the assembly and decompile it to see the code if you are really worried.

http://www.red-gate.com/products/dotnet-development/reflector/

Another alternative is to essentially test the assembly and run some unit-type tests against it.

Stefan H
  • 6,635
  • 4
  • 24
  • 35
  • In the case of 3rd party code being incorporated into a program by the OP while writing the software, I think detecting maliciousness in this manner is unlikely to be fully reliable. Further, if "this code might be malicious" is a serious consideration for code you are grabbing yourself, you are doing something very, very wrong. Though this approach is a bit more reasonable if the concern is that the 3rd-party code might be poorly written (or that the OP might choose to replace it with his own code in the future). – Brian Dec 22 '10 at 14:11
  • I suppose that for the situation where the code is just poorly written, some heuristic analysis might be better than nothing. I'll consider this as an alternative if nothign else surfaces – Robert Dec 22 '10 at 19:09
0

If you debug your application in visual studio and disable "Just my code" you will be able to see the recursive calls in the "call stack". That is the general recommendation for finding root cause of stackoverflow exceptions in .net

basarat
  • 261,912
  • 58
  • 460
  • 511
  • I'm not trying to find stackoverflow exceptions at compile/debug time, I'm trying to detect them at runtime. – Robert Dec 22 '10 at 19:16
  • The only solution (one used by IIS as well :)) is to load the program in a separate appdomain. The at is the *only* way known to me and people around me. – basarat Dec 23 '10 at 04:01
0

Well, running the hostile assembly in a separate thread may reduce the risk of this kind of issue. However, a hostile thread may be difficult or impossible to shut down, so you would be better off running the hostile code in a separate process (hosted by your own code). This gives you a few abilities:

  1. Since you are hosting the hostile code, you can put security restrictions on it.
  2. The code can be run at lower than the default priority, making it less likely to lock up your program or the OS itself.
  3. If anything goes wrong, you can just kill the process.

All that being said, I generally don't consider Denial of Service attacks and the like to be a major concern when loading untrusted code, since if evil code is causing my application to stop working, I'll just stop using the evil code. It's when the evil code is doing something really dirty that I am not aware of that I would get concerned.

Finally, there's always the nuclear option of not allowing users to attach C# code to your application. You could always just use your own special scripting language to give the user access to any parts of your application they need.

Brian
  • 25,523
  • 18
  • 82
  • 173