2

I have a java program which just reads a 12GB file 

 Reader reader1 = new FileReader(filePath);
        try (BufferedReader bufferedReader = new BufferedReader(reader1)) {
            String line = bufferedReader.readLine();
            while (line != null) {
                line = bufferedReader.readLine();
            }
        }

It takes around 53 seconds to read the file for the first time.

Then i checked if the file is cached or not using RAMMAP, the file is completely cached and it is in standby list.

Again i ran the above program, this time it took around same 53 seconds. Why does the performance not improved even when the file is completed cached by OS buffer.

I also created a RAM DISK, copied the input file in the RAM DISK, then again ran the above code, even now it took around 54 seconds. When RAM IOPS is lot more faster, why it is not reflecting here

i0707
  • 617
  • 1
  • 6
  • 14
  • Please try running that program after fresh restart of OS. File might have been cached even on the first attempt. Other option is that your HD is fast enough for sequential read operations and it is actually java readLine processing which is a bottleneck - please see cpu utilization of the java program while you are running it. – Artur Biesiadowski Apr 27 '16 at 10:04
  • @ArturBiesiadowski, I cleared the file from cache using RAMMAP. CPU utilization is between 4 - 6%, it is a 2 processor and 24 core. – i0707 Apr 27 '16 at 11:32
  • Seems that CPU is a bottleneck. Try maybe to split file and load it by multiple threads in parallel? – gudok Apr 27 '16 at 14:21
  • CPU is not a bottle neck, it is a 2 processor and 24 core, the utilization never goes above 6% – i0707 Apr 27 '16 at 14:28
  • Your task is purely sequential (except for GC). You could run 100 CPU with 100 cores each with zero gain. The utilization does not show that one core is 100% utilized; it only shows that most cores have nothing to do. – maaartinus Apr 27 '16 at 18:18

2 Answers2

2

Prefetching.

You're reading the file sequentially. Your disk is fast enough to deliver 12GB / 53s = 226 MB/s. After reading a part of it, the OS assumes you'll need more and more and reads next chucks into memory so that they're available when needed.

The speed is the maximum of needed CPU and needed IO time. The CPU needs 53s, the disk is not slower. You task is CPU-bound.

When RAM IOPS is lot more faster, why it is not reflecting here.

Read 12GB from random positions or from many files, and you'll see how slow a disk can be. In your case, the duration computes as a maximum and max(53, 1) == max(53, 53).

maaartinus
  • 44,714
  • 32
  • 161
  • 320
  • even if the file is read sequentially, why there is no difference in performance between RAM DISK and Hard Disk. As you said my HDD is delivering 226 MB/s, but my benchmark with RAM DISK gives 8 times higher speed than the HD. Is it because of some restriction in java, which doesn't make use of RAM DISK fully? – i0707 Apr 27 '16 at 14:25
  • can you elaborate this " the duration computes as a maximum and max(53, 1) == max(53, 53)." – i0707 Apr 27 '16 at 14:26
  • 1
    @i0707 Due to the prefetching, the operations can (nearly) perfectly overlap. You program needs 53 seconds worth of CPU and less time worth of IO. It doesn't matter, whether the IO needs 53 seconds (quite possible with plates) or 1 second (quite possible with the ramdisk). You always need at least 53 seconds (you'd more if the HD would provide less than 226 MB/s). That's my max-equation. The times don't get added, maximum is right. – maaartinus Apr 27 '16 at 18:14
1

12GB that's 12,000,000,000 bytes of data. Assuming a line in your file is 100 bytes on average that's 120,000,000 lines. Which means 120 million iterations of your loop. 53 seconds looks pretty good doesn't it?

There are 120 million instances of line that need to be garbage collected. There are 120 million calls to readline and 120 million memcpys. 53 seconds seem fair enough and this looks like a bit of code that isn't completely IO bound.

And BufferedReader is slow. Really slow. See this answer: Why is the performance of BufferedReader so much worse than BufferedInputStream?

Community
  • 1
  • 1
e4c5
  • 52,766
  • 11
  • 101
  • 134
  • 1
    even though it is faster, then what is the purpose of caching the file by the OS (windows), if caching is meant for performance boost, then in this case, there must be at least some improvement, which didn't happen – i0707 Apr 27 '16 at 11:33