2

Add into array list more than 50,000,000 records from data base. I am adding 30,000,000 records it showing out of heap memory error .

Developer Marius Žilėnas
  • 5,813
  • 1
  • 41
  • 70
Raghu
  • 21
  • 1
  • 2

5 Answers5

2

Have a look here: Increase heap size in Java

You can set a flag on the JVM to increase the heap size. Use whatever you want but be sure you have the RAM obviously.

From the command line:

java -Xmx8g myprogram
java -Xmx16g myprogram

Or if you know how much space your program needs, set the initial heap size:

java -Xms4g myprogram

Use these cautiously. As others have commented, this is more likely a problem with your approach than the default limit of the Java heap.

Community
  • 1
  • 1
Ron
  • 1,450
  • 15
  • 27
  • 1
    Also worth sizing the ArrayList initially (`new ArrayList<>(50_000_000)`) if the final size is known. – assylias Jan 16 '14 at 11:16
  • @assylias I won't even get started on code optimizations for this use-case. That's a whole different question. – Ron Jan 16 '14 at 11:17
  • 1
    I'm just saying that if you don't do that, then an arraylist with 50m elements will have a capacity of 70m (just tested) and will hence unnecessary allocate an `Object[20_000_000]` which will take around 80MB of memory to store null references... – assylias Jan 16 '14 at 11:19
  • @assylias While that may be true, your optimization reduces the space complexity by around 30%. While the space complexity should really be reduced by 99.99999% – Ron Jan 16 '14 at 11:22
  • Yes I agree with that. – assylias Jan 16 '14 at 11:23
1

Use chunking, as it will increase performance and solve your heap memory error. I mean use 100 arralist of 50000 size, in total it will be 5000000.

Priyank Gupta
  • 803
  • 2
  • 8
  • 10
  • 2
    ? 100 arraylists of 50k elements and a arraylist of 50m items will use more or less the same amount of space. – assylias Jan 16 '14 at 11:15
  • space may not be the problem but it will give problem when you iterate through it, I have encounter the similar problem and followed the above said approach – Priyank Gupta Jan 16 '14 at 11:18
  • ArrayList#get is an O(1) operation regardless of the size of the list (it's just a pointer offset). Your comment does not correspond to what an ArrayList is... – assylias Jan 16 '14 at 11:21
  • then the option will be increasing the heap size – Priyank Gupta Jan 16 '14 at 11:27
1

Another consideration is to look at the 50000000 records and see if they can be logically grouped into smaller groups. Then, process each group by itself so you don't run out of memory. Example: All the names in a telephone book can be grouped by the first letter of last name. Therefore all people who's last name begins with 'A' are grouped together and processed first, moving on to last names beginning with 'B'. I think this will a better approach than arbitrarily processing a group of 10,000 records at a time.

If you do decide to put the data in a database, you should carefully design the database schema and not just create a single database table to hold all 50,000,000 records.

user2810910
  • 279
  • 1
  • 2
0

Increasing your heap size is one option, as Ron points out. Normally when there are a lot of records to be processed, the best practice is to determine a way to process them sequentially so that only one record plus whatever summary data is needed has to be in memory at the same time. In addition, if I need to find the max of a column in a table, or an average of it, I can use a query, etc.

Teresa Carrigan
  • 668
  • 8
  • 14
0

You shouldn't store so much data an ArrayList. For such reasons where databases invented.

If you want to get a specific data you should execute a query and then store the result as an object, and put these objects into an a List if you need them any more.

If you need to Aggregate Data you could also make a Wrapper objects for tha data.

Gábor Csikós
  • 2,787
  • 7
  • 31
  • 57