4

I am trying to read a text file and store every line into ArrayList However, the text file is too long (about 2,000,000) lines and error: java.lang.OutOfMemoryError occurs.

How do i know if the arraylist is full and then create another arraylist to store the remaining data automatically?

Sorry for my poor english.

Thanks for your help.

Dennis Cheung
  • 49
  • 2
  • 4
    The problem is not the number of elements in the ArrayList. The problem is the total memory occupied by the Strings. Splitting the ArrayList won't help you. – Eran Nov 29 '15 at 10:15
  • How will that solve your problem at all? – fge Nov 29 '15 at 10:15
  • what do you want to do with all these lines ? treat each one, save them, etc. ? next steps depend on that . 2 000 000 lines is quite big. – guillaume girod-vitouchkina Nov 29 '15 at 10:22
  • `OutOfMemory` means your whole program is out of memory, not the arraylist. You are trying to store more information than your program can hold. You probably need to look at your actual requirements and figure out how to do it without storing all these strings at the same time. – khelwood Nov 29 '15 at 10:26
  • See also [this question](http://stackoverflow.com/questions/7413830/java-read-line-from-file). – Elliott Frisch Nov 29 '15 at 22:16

3 Answers3

1

2 million lines is far beyond the maximum size for Java Collection (INTEGER.MAX_VALUE or 2 billion indexes).

You are more likely to have heap space outOfMemory error. You can do either

  1. Increase your JVM maximum heap memory allocation.

java -Xmx4g

4g = 4GB.

The default maximum heap size is half of the physical memory up to a physical memory size of 192 megabytes and otherwise one fourth of the physical memory up to a physical memory size of 1 gigabyte.

http://www.oracle.com/technetwork/java/javase/6u18-142093.html

  1. as konsolas recommends, read line by line and store it into a file and flush the variable.

Hope it helps!

Joshua H
  • 754
  • 8
  • 18
0

This depends on what you are planning to do with the file. You're definitely not going to be able to store all of it in memory, as shown by your error.

Depending on what you're trying to do with the file, processing it in blocks and then saving it would be a better idea. For example:

  • Read the first 1000 lines of the file
  • Process these lines/save into another file, etc.
  • Read the next 1000 lines
  • etc.

An ArrayList can theoretically hold 2,147,483,647 items. (max int)

konsolas
  • 1,041
  • 11
  • 24
0

As the other answers suggested, your problem is because you run out of memory before your ArrayList is full. If you still don't have enough memory after increasing the heap space size, BigArrayList will solve your problems. It functions like a normal ArrayList and automatically handles swapping data between disk and memory. Note that the library currently supports a limited number of operations, which may or may not be sufficient for you.

ElementZero
  • 16
  • 1