1

In my java application, I create threads where large files are copied using FileOutputStream and FileInputStream wrapped in BufferedInputStream and BufferedOutputStream. After a file is copied, I try to call close() for the buffered input/output streams but this takes several minutes on Ubuntu. When this is run on OS X, the streams close immediately. Is there a way to speed up this process?

Kalmar
  • 195
  • 2
  • 18
  • Have you tried `try with resources`? – PM 77-1 Jul 24 '15 at 20:33
  • Are you sure your MAC does have an SSD while your Ubuntu has an HDD? – Peter Lawrey Jul 24 '15 at 20:39
  • If it's just about copying files, then you can use [Files.copy(…)](https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#copy%28java.nio.file.Path,%20java.nio.file.Path,%20java.nio.file.CopyOption...%29) since Java 7, which may solve your problem. Another solution could be to use `FileChannel.transferTo(…)` (see for example [this answer](http://stackoverflow.com/a/1690724/864936)). – Tom Jul 24 '15 at 20:43
  • @PM 77-1 I tried that but it didn't make a difference. – Kalmar Jul 24 '15 at 20:49
  • @PeterLawrey My Mac has an HDD, so I guess that's not the reason it's faster. – Kalmar Jul 24 '15 at 20:49
  • @Tom I need to be able to capture the current bytes transferred so that I can update a progress bar. I believe I tried Files.copy() in the beginning and I wasn't able to get the bytes for the progress. – Kalmar Jul 24 '15 at 20:49
  • @Kalmar in that case it is unlikely the Mac is actually writing more than about 40-80 files per second to disk, but the OS is not blocking the process from closing. – Peter Lawrey Jul 24 '15 at 20:50
  • 1
    It sounds like a difference in buffering policy. I don't know about OS X, but on Linux, writing to a file doesn't actually write to disk, it just stores the data in memory to be written in the background (unless the fs is mounted with `-o sync`). However, I didn't think `.close()` would normally wait for the data to reach disk in the way that `fileoutputstream.getFD().sync()` would. How is the total copy+close time on both platforms? – that other guy Jul 24 '15 at 20:57

1 Answers1

0

On linux, a close() attempts to commit the data to disk. This is much, much faster if you have an SSD (as much as 1000x faster) if you have a spinning disk you can write to about 40 files per second (even if very small) If you have an SSD this would be about 20,000 per second on a laptop.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • Do you have a source for this? The close syscall does not do this, and I straced a java test program and did not see a fsync. – that other guy Jul 24 '15 at 20:41
  • @thatotherguy it's not a full fsync, it the updating of the inode and directory, rather than the file itself. Have you used `atop` or `iostat` to see the rate of IOPS you are performing? – Peter Lawrey Jul 24 '15 at 20:43
  • 1
    My Mac actually has an HDD – Kalmar Jul 24 '15 at 20:50