This is almost certainly a JIT problem.
It is worth noting that StackOverflowError
's are triggered in a very odd way due to the fact that the "stack" in java is not very conventional. I won't go too deep into the details, but it appears to essentially be a linked list in the heap.
Given some of the symptoms people have reported in the comments, I think it is strongly related to the JIT. To quote Simon
When I call the method in a simple for (int i = 0; i < 100000; ++i) { fib(9000); } one of two cases appears: either the program crashes on its first call of fib(9000) or on none of the 100000 calls.
Generally the JIT should be activated at a consistent point, especially if there is no variance in the run. If you decide to turn on compilation printing, you will notice that a lot of classes are being compiled that don't seem to relate to your program at all. This could be problematic in the sense that it might take a while for your code to get compiled (JIT compilation is handled as a queue). Another issue is that the JIT is non-blocking. What this means is that the JIT will be triggered, then it will compile the method in another thread, then it will inject the method pointer into the program. Because it is non-blocking, it may finish after the function has triggered a StackOverflowError
.
There are ways that we could improve the odds of success.
- Warm up the method by calling it many times with a sizeable number that never triggers a StackOverflowError. This should give the JIT time to inject the compiled method.
- Set the program to run in single core mode. This can be done (I can only confirm how to do this in Win10) by opening Task Manager, right clicking on the Java process, then clicking on
Go to details
. This will bring you to the Details page with the process selected. You can now right click the process detail and select Set affinity
, deselect all cores but one. This should force the program to run in a single core context, which will cause all threads to be blocking, including the JIT. This does have the notable caveat that the GC will now be blocking.
- Follow the same steps as before to get to the Details page, right click the process detail and select
Set priority
, then select High
(I have found that selecting Realtime
tends to stall the system). This will put that processes thread at the top of the list for other things to run, this will hopefully allow your JIT to get more continuous time on the core for compilation.
- Put the program onto a minimal linux server, this will be some what like #3. The goal here is to have nothing else running on the server. This isn't really recommended, but it may work.
For #2 and #3 you may have to have something in your program to stall execution until you set the priority or affinity. Optionally you could use a program like Process Hacker to save the prioritization or affinity and have it set immediately at every launch.