Products such as GlassFish, JBoss, WebSphere, ... "simply" make use of the Java class loading mechanism to create isolation. By using multiple class loaders, even a static class fields could exist "more than once"; and each "domain" gets its very special version of that.
Start reading here, or there for example:
Application Universe – Each Java EE application has its own class loader universe, which loads the classes in all the modules in the application.
And beyond that, look into this.
In other words: although the System class obviously represents a "system view" - the class loading mechanism should make it possible to give different System class instances to each domain. Thereby making it possible to have domain-specific properties within each domain-specific System class. But to be precise: I could not find clear evidence to back up this statement. On that topic, here and there should help (the later one indicating that there even ways to tamper with the system classloader).
But thinking further, there is a problem with that idea: there are actually two class loading mechanisms. There "system/user" class loaders ... and the initial bootstrap classloader.
And the bootstrap classloader is actually "baked" into the JVM (it is implemented in native code for example) - and it can't be replaced.
And java.lang.System should be loaded by that bootstrap classloader!
So it is impossible to use "classloader magic" to enable "per domain" System properties!
The other option I see is to create a Java agent that intercepts calls and manipulates their result (see here as starting point).
But again; agents come "after" bootstrapping has completed!
So the only logical conclusion (taking what is left after excluding all options): the premise of the question must be wrong!
It is simply impossible that different "applications" running on the same JVM give different system properties. And luckily, the great answer by Seelenvirtuose confirms that conclusion.