Sandbox Java code before compiling?

Question

If there is a system where the client writes Java code (*.java files), and submits them to a server running in Java, how can I make the submitted code not do unwanted and malicious things?

For a little more depth: The client would write the plain text Java code (their classes), and the code would be sent to the server. On the server, the code would be inspected to make sure it only does allowed actions. If it passes the test, it gets compiled into a .class file, where it is then loaded with Java's class loader. Finally, using Reflection, methods can then be called and passed parameters.

I think I can see how to do most of that, but how do I make sure code that's compiled is "safe"? For example, I wouldn't want it to spawn threads, or perform file IO, or interact with the system it's running on in any way. I was thinking that I could scan the input file for imports, and if they are not on a white list, reject the file. On top of that, I was thinking that if I did want to limit something, like say the number of threads, I could allow the import of a proxy class that would keep track of threads per user.

Would that work, or if not is there a way?

Answer 1

What you are trying to do would be a never ending battle with people who choose to do malicious stuff. I would therefore recommend that you look at this page on how to sandbox your code.

http://securesoftwaredev.com/2012/11/12/sandboxing-java-code/

To summarize

• By default, Java runs without a SecurityManager, so you should add code to your application to enable one:

  System.setSecurityManager(new SecurityManager());
  

• A permission represents access to a system resource.

• Use Code signing

To assemble the pieces in a sandbox:

1. Install a SecurityManager
2. Sign the application jars
3. Grant all code signed by us AllPermission
4. Add permission checks in places that code may call
5. Run the code after the permission checks in a doPrivileged() block

Answer 2

Java has some sandboxing mechanisms, but those have such a poor security track record that many recommend turning off Java in the browser entirely.

You could try sandboxing a VM inside it's own virtual OS with reduced privileges.

Alternatively, you could require the code to be run in a language like Joe-E that is designed to allow Java programs to execute untrusted code.

Joe-E is a subset of Java that makes it easier to architect and implement programs with strong security properties that can be checked during a security review. It enables programmers to apply the principle of least privilege to their programs; implement application-specific reference monitors that cannot be bypassed; introduce and use domain-specific security abstractions; safely execute and interact with untrusted code; and build secure, extensible systems. Joe-E demonstrates how it is possible to achieve the strong security properties of an object-capability language while retaining the features and feel of a mainstream object-oriented language.

The degree to which Joe-E's designers had to depart from standard Java should give you an idea of how big a task it is to prevent untrusted Java code from abusing the ambient authority available from within a JVM.

---

While some of these approaches may protect you from abuse of authority (especially if layered together), none will prevent denial of service. If you're trying to run other people's code alongside your code, and it decides to try to take all the CPU and never give it back, your only option is often to kill the whole process.

Answer 3

White listing is the option. with blacklisting, attackers will find something malicious that is not black listed.

SecurityManager does the same by whitelisting the permissions. This is the way to go.

Code scanning may achieve the same at a different level, but you will reinvent the wheel(securitymanager)

Answer 4

Online judges do this. Some are open source. See if you can leverage either one of these or the code one is developed on. Googling, this product is open source and says the backend is in bash, which may help you with OS-interfacing security.

This judge looks like some random project that may or may not work or implement security, but it's Java. Keep Googling.

Answer 5

I have looked into this matter. The standard Java security model are for each function that may do something dangerous, include code that checks the security privileges of the caller and deny access from someone within the sandbox. Additionally JNI can be disallowed in sandboxes code, for obvious reasons.

The problem with this approach are of course that sometimes developers forget to encapsulate their functions in security checks. Sometimes they do not even realize they create an insecure API and expect its dependencies to in turn deny access from sandboxed applications.

Another problem with this approach are that checking security violations within every API call are quite costly. In the past Java had other performance issues that made this quite low priority. For example the cost of a JNI call are usually so expensive that wrapping it in security code probably do very little to degrade performance. But modern JVM:s are so fast that this is probably becoming one of the mayor remaining bottlenecks.

I do not know how .NET implements sandboxing in modern versions. But I do i know how Mono and Native Client does it. Instead of having sandboxing within framework code they use pre-execution code inspection. They can then generate a class that only use secure API:s before it:s handed to the JIT. This of course ad a small delay to class loading but ends up more secure than the Java security model, and as a bonus there are a performance gain from now running a secure class without any sandboxing.

There are Java libraries that can be used to implement something similar in Java, such ASM and BCEL

The only attempt I can find to use code inspection in Java security are Mobile Code Security Through Java Byte-Code Modification which utilizes BECL for this.