1000000000 array input in java

Question

I am trying to test an algorithm that needs 1,000,000,000 array input.

Scanner scanner = new Scanner(new File("999999998.txt"));
int[] tall = new int[1000000000];
int i = 0;
while (scanner.hasNextInt()) {
    tall[i++] = scanner.nextInt();
}

This exception is thrown:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Do you really need to have all the numbers in memory at once? That's 4 GB right there... — Matti Virkkunen, May 15 '15 at 02:56
yes i need test my algorithm with big input to compare it with else . how can i increase heap? — mojtaba omid, May 15 '15 at 03:01
Specify your algorithm, that will greatly assist in providing more targeted solutions beyond the "if this, do A, otherwise if that, do B, ...". — paxdiablo, May 15 '15 at 03:10

paxdiablo · Answer 1 · 2015-05-15T03:06:07.090

You're trying to create a (roughly) 4G array and it's too big to fit in the heap (one billion 4-byte integers).

Most likely, if you want to do it that way, you're going to need a 64-bit Java running on a 64-bit operating system (and possibly a large amount of physical memory for performance), and increasing the heap size to something much larger than the default (such as with java -Xmx6g or something similar).

Or, if your algorithm is able to operate on the data in sections, that may be a better option.

So, if you're summing the items in the file, you can bring them in a thousand at a time to add them to a running total. Now that's not going to be easy if there's a lot of random access of all sorts of different integers but, in that case, you could possibly create the array on disk and use a caching/LRU interface to ensure you only load in what's needed at any given point.

In addition to a 64-bit JVM and a large heap, depending on the algorithm's access patterns it may also need more than 4GB of physical memory. — Patricia Shanahan, May 15 '15 at 03:03
@Patricia, not sure that's _needed,_ the OS can probably handle paging of the large virtual address space to a limited real address space. However, for performance, it's probably essential so I've made the change. Thanks. — paxdiablo, May 15 '15 at 03:08

blazetopher · Answer 2 · 2015-05-15T03:57:37.660

3

Firstly, I agree with all the comments and answers suggesting that you try to hold on to as little data as possible.

Presuming you really do need all that data, you need to start your java container with more memory using the -Xmx flag:

java -Xmx6g <your launch args here>

And if you can't launch Java using the above args, then it means you either are running 32 bit Java or you don't have 6GB of memory available (which means in both cases that you cannot allocate an array of this size).

edited May 15 '15 at 03:57

answered May 15 '15 at 02:59

blazetopher

1,050
9
13

Thanks @TimBiegeleisen - was revising my answer at the same time you were!! :) – blazetopher May 15 '15 at 03:01
where should i write that code? i tried to write something like that on VM options on netbeans but it dose not work – mojtaba omid May 15 '15 at 03:08
Look here: http://stackoverflow.com/questions/15460779/how-to-increase-the-java-heap-size-in-netbeans – Tesseract May 15 '15 at 03:15

score 0 · Answer 3 · answered May 15 '15 at 03:26

I would recommend against an almost 4GB array for many reasons, first you will eat most, if not all your ram resources and second its probably easier, and overall better to break it down.

Try to break up the data into different files or simply read the data from the file into the array piece by piece and handle it like that.

If you want handle the data like this regardless do the following:

1). Make sure you a 64bit operating system and a 64bit java version or this will be impossible.

2). Increase your java memory heap size to over 4GB maybe 5 or 6. Use the command java -Xmx6g programName where xmx is the command, 6g is is heap size and programName is the name of your program. When running your program from windows command prompt. Or just set in your IDE.

Hope this helps.

1000000000 array input in java

3 Answers3