Fastest method for reading file data in Arrays (Java)

Question

Take a look at the following link:

http://snippetsofjosh.wordpress.com/tag/advantages-and-disadvantages-of-arraylist/

This one of the reasons why I always prefer to use Arrays instead of (Array)Lists. Still, this got me thinking about memory management and speed.

Hence I arrived at the following question:

What is the best way to store data from a file when you don't know the size of the file (/number of entries) (where best is defined as 'the least amount of computation time')

Below, I will present 3 different methods and I would like to know which one of them is best and why. For the clarity of the question, let's assume I must end up with an Array. Also, let's assume every line from our .txt file only has one entry (/one string). Also, for limiting the scope of the questions, I will limit this question to Java only.

Let's say we want to retrieve the following info from a file called words.txt:

Hello
I 
am
a
test 
file

Method 1 - Double and dangerous

File read = new File("words.txt");
Scanner in = new Scanner(read);

int counter = 0;

while (in.hasNextLine())
{
    in.nextLine();
    counter++;
}

String[] data = new String[counter];

in = new Scanner(read);

int i = 0;

while (in.hasNextLine())
{
    data[i] = in.nextLine();
    i++;
}

Method 2 - Clear but redundant

File read = new File("words.txt");
Scanner in = new Scanner(read);

ArrayList<String> temporary = new ArrayList<String>();

while (in.hasNextLine())
{
    temporary.add(in.nextLine());
}

String[] data = new String[temporary.size()];

for (int i = 0; i < temporary.size(); i++)
{
    data[i] = temporary.get(i);
}

Method 3 - Short but rigid

File read = new File("words.txt");
FileReader reader = new FileReader(read);

String content = null;

char[] chars = new char[(int) read.length()];
reader.read(chars);
content = new String(chars);

String[] data = content.split(System.getProperty("line.separator"));

reader.close(); 

If you have an alternative way (which is even better) then please supply it below. Also, feel free to adjust my code where necessary.

---

Answer:

The fastest method for storing data in an array is the following method:

File read = new File("words.txt");
Scanner in = new Scanner(read);

ArrayList<String> temporary = new ArrayList<String>();

while (in.hasNextLine()) {
    temporary.add(in.nextLine());
}

String[] data = temporary.toArray(new String[temporary.size()]);

And for Java 7+:

Path loc = Paths.get(URI.create("file:///Users/joe/FileTest.txt"));
List<String> lines = Files.readAllLines(loc, Charset.defaultCharset());
String[] array = lines.toArray(new String[lines.size()]);

Answer 1

I assume that best means faster here.

I would use method 2, but create the array with the methods provided by the Collection interface:

String[] array = temporary.toArray(new String[temporary.size()]);

Or even simpler (Java 7+):

List<String> lines = Files.readAllLines(file, charset);
String[] array = lines.toArray(new String[lines.size()]);

Other methods:

• method 1 does two passes and it is very unlikely that reading a file is more efficient than resizing an arraylist
• I am not sure if method 3 is faster or not

---

Update:

for the sake of completeness, I have run a microbenchmark with the modified method2 as above and including an additional method (method4) that reads all bytes at once, creates a string and split on new lines. The results (in mn microseconds):

Benchmark   Mean 
method1     126.178
method2     59.679
method3     76.622
method4     75.293

---

Edit:

with a larger 3MB file: LesMiserables.txt, the results are consistent:

Benchmark      Mean 
method1     608649.322
method2      34167.101
method3      63410.496
method4      65552.79

Answer 2

A very good comparison with all the source code is given here java_tip_how_read_files_quickly

Summary:

For the best Java read performance, there are four things to remember:

• Minimize I/O operations by reading an array at a time, not a byte at a time. An 8Kbyte array is a good size.
• Minimize method calls by getting data an array at a time, not a byte at a time. Use array indexing to get at bytes in the array.
• Minimize thread synchronization locks if you don't need thread safety. Either make fewer method calls to a thread-safe class, or use a non-thread-safe class like FileChannel and MappedByteBuffer.
• Minimize data copying between the JVM/OS, internal buffers, and application arrays. Use FileChannel with memory mapping, or a direct or wrapped array ByteBuffer.

Hope that helps.

EDIT

I would do sth like that:

File read = new File("words.txt");
Scanner in = new Scanner(read);    
List<String> temporary = new LinkedList<String>();

while (in.hasNextLine()) {
    temporary.add(in.nextLine());
}

String[] data = temporary.toArray(new String[temporary.size()]);

The main difference is reading data only once (as opposed to other 2 methods) and addition in linkedlist is very cheap + no extra operation on lines needed (like splitting) - don't use arraylist here

Answer 3

If you are reading data from a file, the bottleneck will be the file reading (IO) stage. The time spent processing it will be insignificant in almost all cases. So do what is correct and safe. First you make it right; then you make it fast.

If you don't know the size of the file you must have some kind of dynamically expanding data structure. Which is what ArrayList is. Code you write yourself is unlikely to be more eficient or correct than such an important part of the Java API. So just use ArrayList: option 2.

Answer 4

I would use guava

File file = new File("words.txt");
List<String> lines = Files.readLines(file, Charset.defaultCharset());
// If it really has to be an array:
String[] array = lines.toArray(new String[0]);

Answer 5

List<String> lines = Files.readAllLines(yourFile, charset);
String[] arr = lines.toArray(new String[lines.size()]);