2

I have a program in which I want to allocate the largest array possible. Is there a way in Java to query the VM to find out what that is? If so, how can I do it?

My latest attempt at getting the information was trial and error: I started trying to allocate the array with size Integer.MAX_VALUE and kept decrementing the size by 1 until the allocation was successful. But, it's taking a very long time; apparently the actual maximum is nowhere near the theoretical limit. I did establish that it's at least Integer.MAX_VALUE + 1. Come to think of it, it's at least 1.25 billion bytes.

Bruce Feist
  • 621
  • 5
  • 10
  • The Runtime class has some methods you may find useful...https://docs.oracle.com/javase/8/docs/api/java/lang/Runtime.html – Eric Jul 19 '17 at 17:15
  • Trying to calculate how many `int`s there should be room for based on the amount of memory you provide the JVM will **always** give you the wrong amount. The JVM uses a decent amount of memory itself to init everything. If you profile an application that just has a `while(true)` loop you can see that there are over 1.5k classes loaded just to get the thing running – CraigR8806 Jul 19 '17 at 17:16
  • Why would you want to create an array that big? ArrayList is not much slower than an array and will grow to meet your needs. – bhspencer Jul 19 '17 at 17:16
  • Further Info: https://stackoverflow.com/questions/2015463/how-to-view-the-current-heap-size-that-an-application-is-using – Eric Jul 19 '17 at 17:20
  • wrt. actual vs theoretical number: to increase the actual limit, adjust also the max heap for your JVM by `-Xmx16G`. Also `byte[] array` will obviously take less space than `double[] array`, actually 8 times less. – Jiri Kremser Jul 19 '17 at 17:25
  • @Eric I didn't find anything that helped me with this specific problem, but I did find availableProcessors(), which will be useful for me. – Bruce Feist Jul 19 '17 at 17:39
  • @bhspencer ArrayLists are based on arrays, and have the same maximum size -- no help. – Bruce Feist Jul 19 '17 at 17:45
  • @Jiri I'm using byte[], and I've already increased the heap to 3G. – Bruce Feist Jul 19 '17 at 17:47
  • @BruceFeist but why do you need to know the max possible size of an array? If you create the biggest array you can you will run out of memory almost immediately afterwards? What possible use can this have? – bhspencer Jul 19 '17 at 18:39
  • @bhspencer I explained that the use in my response to Kayaman's suggested answer below. You generally won't run out of memory immediately after allocating such an array; the theoretical maximum size of an array in Java is Integer.MAX_VALUE bytes, which can be a good bit smaller than the total amount of memory that Java has to work with. – Bruce Feist Jul 19 '17 at 18:45

3 Answers3

2

The maximum theoretical array size is Integer.MAX_VALUE (as the internal VM structure holds the length in an int), but Hotspot didn't at least at one point allow arrays larger than Integer.MAX_VALUE - 5.

However that's still a lot of memory. Of course it's hard to imagine a sensible program allocating an array "as big as I can", since it would start to be harmful to performance. The GC likes to move things around, and it's tuned for objects and not huge chunks of memory.

You could of course check the free memory (with Runtime.freeMemory() etc.) and allocate based on that, but I'd bet that whatever problem you're trying to solve can be done in a lot better way.

See also Do Java arrays have a maximum size?

Of course the MAX-5 is for Hotspot. Other VMs could support something else, but it probably wouldn't make much of a difference.

Kayaman
  • 72,141
  • 5
  • 83
  • 121
  • The program is my attempt to see how many prime numbers I can find, and how quickly. The array is being used as a bit array storing 8 booleans in each byte; each boolean represents whether or not an odd number is prime. I've run it with an array of 10 billion bits. Since that's bigger than Integer.MAX_VALUE, it's indexed with a long instead of an int. – Bruce Feist Jul 19 '17 at 17:33
  • That would probably warrant a bit more advanced approach, with multiple arrays. For additional speed (array access does bounds checking) the `Unsafe` class can give you direct, faster access to the array(s). – Kayaman Jul 19 '17 at 17:38
  • Surprisingly enough, speed isn't an issue at the moment; the program finds all primes up to 10 billion in a couple of minutes. I'm not familiar with the Unsafe class, and I don't see it in the documentation. – Bruce Feist Jul 19 '17 at 18:00
  • It's not for general usage (although I think it was planned to be in Java 9), but when you really need speed and are willing to go through some trickery, it can be used to improve performance in certain situations. Some [high speed Java libraries](https://github.com/OpenHFT) use it when speed is of the essence. Anyway, a class that contains multiple large size arrays could be used to provide a large "pseudo array". – Kayaman Jul 19 '17 at 18:12
2

In java.util.ArrayList maximum array size is defined as Integer.MAX_VALUE - 8 (as of Oracle JDK 8):

/**
 * The maximum size of array to allocate.
 * Some VMs reserve some header words in an array.
 * Attempts to allocate larger arrays may result in
 * OutOfMemoryError: Requested array size exceeds VM limit
 */
private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;
Vladimir Petrakovich
  • 4,184
  • 1
  • 30
  • 46
  • I'll give it a try, but my program should have figured that out quickly if it's that close to Integer.MAX_VALUE. – Bruce Feist Jul 19 '17 at 17:35
0

It's not elegant, but I've figured out a way of getting the information I need. I mentioned above a sequential search that I tried to find the largest possible byte array with; it was far too slow. I switched it to a binary search, and now it completes in about 7 seconds. Not very elegant, and takes significant time, but it's fast enough and it works. Here's the code fragment that does the job:

      int can = 0, cant = Integer.MAX_VALUE;

      do {
         int middle = (int) ((can + (long) cant) / 2); // long is to avoid overflowing in the sum
         allocated = false;
         try {
            byteArray = new byte[middle];
            allocated = true;
         }
         catch (java.lang.OutOfMemoryError e) {
            byteArray = null;
         } // catch

         System.out.printf("Trying to allocate %,d bytes\n", middle);

         if (allocated)
            can = middle;
         else
            cant = middle;
      } while (cant - can > 1);

      if (! allocated) // happens if middle failed to allocate in the last iteration above
         byteArray = new byte[can];

On my system, I was only able to allocate 1,386,916,521 bytes.

Bruce Feist
  • 621
  • 5
  • 10
  • You do realize that your process for finding the max array size is using memory along the way reducing the size of the array you can allocate? – bhspencer Jul 19 '17 at 18:41
  • Yes, but it should be garbage collected since I'm not keeping any references to it. But that's an interesting thought; maybe I should invoke gc() explicitly after each successful allocation. – Bruce Feist Jul 19 '17 at 18:47
  • It is not possible to explicitly invoke the gc in Java. All you can do is indicate to the gc that you would like a collection at some point with System.gc(); – bhspencer Jul 19 '17 at 18:49
  • @bhspencer But isn't that always the case? – Bruce Feist Jul 19 '17 at 21:33