Let's see...
An int
occupies 32bit or 4 bytes in memory.
Your 2-dimensional array thus requires
10000 * 10000 * 4 bytes = 400000000 bytes = 381.47 MB
(plus storage for the array objects)
Let's verify that with a simple test:
int n = 10000;
int[][] foo = new int[n][n];
System.out.println("USED MEMORY: " + (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()) / 1024 / 1024);
Result: 384 MB (on my machine, your results may vary).
Running out of memory in the heap happens when a requested block of memory cannot be allocated, not necessarily when the memory is full.
As you allocate memory in blocks of 40'000 bytes, you can run out of memory when no contiguous free memory of that size is available (hundreds of blocks with less that 40'000 won't help here). That means you can easily run out of memory when the free memory on the heap is fragmented as small memory blocks.
If you happen to allocate that memory multiple times in a row, try reusing the allocated memory instead of allocating it again (to prevent fragmentation to a certain degree).
Alternatively, try using smaller arrays (even if it means using more of them).