How good is a try catch at catching out of memory exceptions?
It works just fine at one level ... but at another level it can be risky, and / or futile. See below.
Is it possible that the program managing memory runs out of memory?
The "program" managing memory is the garbage collector. AFAIK, it always has enough memory for its own purposes. Though if it didn't it would have no alternative but to hard crash the JVM.
When could I try to catch an out of memory exception and it would fail to catch the exception?
Only if the JVM crashes. Or, if the OOME is thrown on a different thread stack to the one where you are trying to catch it.
OK, so why is catching OOME risky and / or futile.
The first reason is that an OOME can be thrown on any thread stack without any warning. When you catch the exception, the handler (typically) has no way to know what was happening "up stack", and exactly how it failed. Hence, it has no way of knowing if the application's execution state has been damaged or compromised. (Was the thread in the middle of updating something important? Was it about to notify some other thread ... and the thread will now never get the notification?)
The second reason is that OOMEs are often a result of a Java storage leak ... caused by something "hang onto" objects when it shouldn't do. If you catch an OOME and attempt to resume ... when the problem was caused by a leak ... the chances are that the offending objects will still be reachable, and another OOME will follow pretty soon. In other words, your application could get stuck in a state where it is continually throwing and recovering from OOMEs. At best this is going to kill performance ... because the last thing that the JVM typically does before an OOME is to perform a full (stop the world) garbage collection. And that takes a significant time.
Note that is not to say that you should never catch OOMEs. Indeed, catching an OOME, reporting it and then shutting down is often a good strategy.
No, the thing that is risky / futile is to catch the OOME and then attempt to recover and continue running.