Java Out of Memory Array Error

Question

I'm trying to store a large amount of longs into an array, but I get an OOM error with my array and ArrayList.

    ArrayList<Long> A = new ArrayList<Long>();
    Long[] B = new Long[100000000];

    for(int i = 0; i < 100000000; i++) A.add(i);        
    for(int i = 0; i < 100000000; i++) B[i] = (long) i;

//Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
//at java.lang.Integer.valueOf(Unknown Source)

Is there any other simple data structure I can use to store such a large amount of longs or integers?

Answer 1

If you use the -Xmx flag appropriately when running java (see here for more info), you can increase your Heap size. This will allow you to use more memory if needed, in a controlled way. To my knowledge, there is not a way of "asking" for more heap memory from within a program itself (similarly to running the sbrk() or mmap() syscalls in C)

As the answer I linked to says:

For example, starting a JVM like so will start it with 256MB of memory, and will allow the process to use up to 2048MB of memory:

java -Xmx2048m -Xms256m

Also, you can use "k", "m", or "g" for Kilobytes, Megabytes and Gigabytes respectively. You cannot exceed 1GB (Heap size, that is), however, unless you are using the 64-bit JVM.

If you do the math with your particular use-case, assuming 64-bit longs * 100000000 costs about 800MB of space.

Answer 2

Does your program require you to store longs? If you were to use Integers instead of longs then you could store much more, either way, what requires you to store so many longs in an array?

Other solutions:

You could give your program more heap space when you start it with the argument -Xmx2G or some other length greater than the standard 512M or 1G

You could process a less number of array values, then save the array to hard drive. Then process the rest of the array and store it into the file.(Would require basic knowledge of Java's garbage collection)

Answer 3

Any solution depends upon what you're trying to do. For example, if you simply want to iterate through the values, you can declare an Iterator<Long>:

Iterator<Long> myIterator = new Iterator<Long>() {
  private long nextValue = 0;

  @Override
  public boolean hasNext() {
    return nextValue < 100000000;
  }

  @Override
  public Long next() {
    return nextValue++;
  }

  @Override
  public void remove() { throw new UnsupportedOperationException(); }
};

Such a solution has O(1) memory usage, because it doesn't require you to store anything more than the next value. But it may not be appropriate to the application.