How to deal with large arrays in Java

Question

I am reading a file that has 10,000 int values and then trying to store these in an array. There is an exception thrown which says that the array value is too large.

I was wondering, rather than write this array out in to a variable, could i possibly just keep it in memory and read it from there. Would this be a suitable way of solving this problem?

edit:

After more examination it appears that the error being thrown is a "code to large for try statement" error. I am reading each array element and appending it to a string, maybe this is what is causing the error?

10000 is not a large number for an array, but you are better off with a list. — Manoj, Nov 24 '11 at 15:03
Please paste some code which illustrates the issue you are having. A 10000 element array of ints will not cause issues in normal circumstances. — mcfinnigan, Nov 24 '11 at 15:04
Variables are in memory, I'm not sure what distinction you're trying to make. Have you tried increasing the memory available to your application using the -Xmx argument? — Dave Newton, Nov 24 '11 at 15:04
"rather than write this array out in to a variable, could i possibly just keep it in memory" basically same thing. — Bhesh Gurung, Nov 24 '11 at 15:05
I cant use a list because this is data used to invoke a method a user has selected via reflection. I am trying to analyse the run time performance of their method so need to try it with different data sizes. If there method takes an int [] that is exactly what i have to provide it with — Biscuit128, Nov 24 '11 at 15:05

score 3 · Answer 1 · answered Nov 24 '11 at 15:05

An array of 10,000 int values is about 40KB.

You could try to reduce the memory used further however I suspect this is not your problem.

Can you give us the actual error message? An array value is only too large if its a long e.g. say you used File.length()/4 to determine the size of the array, in which case you need to cast it to an int

Michael Berry · Accepted Answer · 2011-11-24T15:28:12.407

You could use an ArrayList instead - but an array should be fine with 10,000 values. Can you post more detail? Code, full stack trace etc. Theoretically it should be fine with Integer.MAX_VALUE elements (a LOT more than 10k), but of course you may run out of memory first!

In terms of "just keep it in memory and read it from there", well variables are just kept in memory, so whether you use an array or a list (or any other data structure) you'll always be reading it from memory!

EDIT: Based on your additional explanation then it's not a problem with the array size at all, it's a problem with you generating 10,000 lines of code to put in a single block, which is too many and thus it complains. Alter your code to generate code that uses a loop instead and all should be well, however many elements you have in there (up to Integer.MAX_VALUE of course.)

score 1 · Answer 3 · answered Nov 24 '11 at 15:06

1

It is strange that you cannot create 10000 elements long array. I believe that your problem is not the array length but the value of particular array element. Anyway if you need bigger arrays use Lists instead. Specifically java.util.LinkedList.

answered Nov 24 '11 at 15:06

AlexR

114,158
16
130
208

Using a `LinkedList` instead of an array would increase the memory required by a factor of at least 3 I'd wager: you have to wrap the `int` in `java.lang.Integer`, then you have the overhead of each entry being represented by an object in the `LinkedList`, so thats at least 2 extra objects per value and Java object headers are at least the size of `int`s (bigger on 64 bit platforms). Never mind the extra work the garbage collector will have to do to deal with so many objects. This just doesn't make sense, sorry downvoting. – Ramon Nov 24 '11 at 15:12

score 1 · Answer 4 · edited May 23 '17 at 10:08

1

Your problem is that you are writing each array or String assignment out in full, something like this:

        array[0] = 0;
        array[1] = 1;
        array[2] = 2;
        // all the way up to 9999!

or this:

        String s = "";
        s += array[0];
        s += array[1];
        s += array[2];
        // all the way up to 9999!

instead of in a loop. So you generate more code than Java will allow in a method.

This results in a compilation error as you describe:

$ javac Test.java
Test.java:7: code too large for try statement
        try {
            ^
Test.java:4: code too large
    public static void main(String[] args) {
                       ^
2 errors

Following from discussion in comments, the code that you say is producing this compiler error does not have an enormous number of lines. Something doesn't make sense - the error you report does not line up with the code you say is causing it. At this late stage I strongly recommend that you post some code, and the error so that others can try to understand what might be causing this.

(Also, your question isn't likely to get much attention because you have accepted an answer. You might want to reconsider that if your question is not in fact answered.)

edited May 23 '17 at 10:08

Community

1
1

answered Nov 24 '11 at 15:34

ewan.chalmers

16,145
43
60

yes this is exactly what ia wrong tho i am not writing the lines as you put, i am doing it in a loop, As i understand it generates the same sort of issue. Just need to do it a little differently now. – Biscuit128 Nov 24 '11 at 15:52
I guess you mean that your are generating the code in a loop - that you are generating the java source file using a loop in a script or similar. – ewan.chalmers Nov 24 '11 at 15:56
forexample; for(int i = 0; i < 10,000; i++){stringbuilder.append(get[i] + " ,");} – Biscuit128 Nov 24 '11 at 16:00
No, that code will not result in the error you report. The error you describe arises from having too much source code, not how many times you iterate in a loop. – ewan.chalmers Nov 24 '11 at 16:08
the methid that throws it is around 60 lines long excluding spaces and catch statements – Biscuit128 Nov 24 '11 at 16:13
There must be an awful lot of characters per line then! Using short lines (like above), I get the compiler error somewhere between 12,000 and 12,500 lines of code. – ewan.chalmers Nov 24 '11 at 16:43
lol! to all extents and purposes its your every day method with normal sized lines :p I am calling this in a swing worker, i am calling method.invoke in there and the code thats loops through a 10,000 sized array building as i showed you before is happening via `Object o = new Object{public String toString(){//code in here}}` `method.invoke(obj, o)` – Biscuit128 Nov 24 '11 at 16:45
can i also add that when the array size is a 1000 it works fine – Biscuit128 Nov 24 '11 at 16:49

score 0 · Answer 5 · answered Nov 24 '11 at 15:06

0

An array of 10,000 ints isn't very big at all. I can't think why you would have a problem keeping the data in memory (ie assigned to a variable).

answered Nov 24 '11 at 15:06

Qwerky

18,217
6
44
80

score 0 · Answer 6 · answered Nov 24 '11 at 15:06

I find it odd that 10,000 ints takes up too much memory. It could be that other stuff if eating up your memory. Have you tried increasing the available memory to Java? (i.e.-Xmx512m). If this is not possible, you can always try to use shorts or bytes if the numbers are small enough.

The array will take just as much space as chunk of memory (like c does).

score -1 · Answer 7 · answered Nov 24 '11 at 15:31

This is a known bug in the JVM. It prohibits you from creating an array of integers with size 10,000 (and also 16,384 on Mac OS X). It has to do with the way in which Java translates the byte code into machine code. Changing the size of the array to 10,001 will solve the problem.

How to deal with large arrays in Java

7 Answers7