A 9 million line file should take less than a few seconds. Most of that time will be spent reading the data into memory. How you split up the data is unlikely to make much difference important.
The BufferedReader and String.split sounds fine to me. I wouldn't use interning unless you are sure this will help. (It won't intern() it for you)
The latest version of Java 6 has some performance improvements in the handling of Strings. I would try Java 6 update 25 to see if it is any faster.
EDIT: Doing some test finds that split is surprisingly slow and you cna improve on it.
public static void main(String... args) throws IOException {
long start1 = System.nanoTime();
PrintWriter pw = new PrintWriter("deleteme.txt");
StringBuilder sb = new StringBuilder();
for (int j = 1000; j < 1040; j++)
sb.append(j).append(' ');
String outLine = sb.toString();
for (int i = 0; i < 1000 * 1000; i++)
pw.println(outLine);
pw.close();
long time1 = System.nanoTime() - start1;
System.out.printf("Took %f seconds to write%n", time1 / 1e9);
{
long start = System.nanoTime();
FileReader fr = new FileReader("deleteme.txt");
char[] buffer = new char[1024 * 1024];
while (fr.read(buffer) > 0) ;
fr.close();
long time = System.nanoTime() - start;
System.out.printf("Took %f seconds to read text as fast as possible%n", time / 1e9);
}
{
long start = System.nanoTime();
BufferedReader br = new BufferedReader(new FileReader("deleteme.txt"));
String line;
while ((line = br.readLine()) != null) {
String[] words = line.split(" ");
}
br.close();
long time = System.nanoTime() - start;
System.out.printf("Took %f seconds to read lines and split%n", time / 1e9);
}
{
long start = System.nanoTime();
BufferedReader br = new BufferedReader(new FileReader("deleteme.txt"));
String line;
Pattern splitSpace = Pattern.compile(" ");
while ((line = br.readLine()) != null) {
String[] words = splitSpace.split(line, 0);
}
br.close();
long time = System.nanoTime() - start;
System.out.printf("Took %f seconds to read lines and split (precompiled)%n", time / 1e9);
}
{
long start = System.nanoTime();
BufferedReader br = new BufferedReader(new FileReader("deleteme.txt"));
String line;
List<String> words = new ArrayList<String>();
while ((line = br.readLine()) != null) {
words.clear();
int pos = 0, end;
while ((end = line.indexOf(' ', pos)) >= 0) {
words.add(line.substring(pos, end));
pos = end + 1;
}
// words.
//System.out.println(words);
}
br.close();
long time = System.nanoTime() - start;
System.out.printf("Took %f seconds to read lines and break using indexOf%n", time / 1e9);
}
}
prints
Took 1.757984 seconds to write
Took 1.158652 seconds to read text as fast as possible
Took 6.671587 seconds to read lines and split
Took 4.210100 seconds to read lines and split (precompiled)
Took 1.642296 seconds to read lines and break using indexOf
So it appears that splitting the string yourself is an improvement and gets you close to treading text as fast as you can. The only way to read it faster is to treat the file as binary/ASCII-7. ;)
