2

I need to write lots of byte arrays sized from 1MB to 2MB to individual files. The data is always allocated as a byte[] (it is calculated) so I cannot use FileChannel.transferTo. After the write it should be guaranteed flushed to the disk.

Is this the fastest solution?

FileChannel channel =  new RandomAccessFile(file, "rw").getChannel();
ByteBuffer buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, data.length);
buffer.put(data);
channel.force(false);
channel.close();

In my test-suite I see that the files sometimes are locked by the OS (or JVM) (java.io.File.delete() returns false) when I do a clean up. I'm concerned about the resources not getting released. People suggest using System.gc() and also some unsafe solutions, but maybe my program could hit a max limit? Should I be concerned? Will it be a problem allocating lots of memory mapped files? It is from the same thread so they will be closed before opening a new. Also my program must be able to delete these files.

Please also read How to unmap a file from memory mapped using FileChannel in java?

Community
  • 1
  • 1
Stig
  • 1,974
  • 2
  • 23
  • 50
  • 2
    Do some benchmarking to see what is fastest. Start with FileOutputStream.write() and see if you can do better with BufferedOutputStream, and so on. I wouldn't jump immediately to mmap'ed files. – President James K. Polk Jan 14 '17 at 16:05
  • Can you check who/what has the files locked? You can always use delete on exit (if the process is not running constantly) – bichito Jan 14 '17 at 16:13
  • I have benchmarked and FileOutputStream + BufferedOutputStream is slower. But I am concerned about the locked files. – Stig Jan 14 '17 at 16:15
  • 2
    A BufferedOutputStream will slow things down if you have all the byte array in memory and can write it all at once directly with the FileOutputStream. – JB Nizet Jan 14 '17 at 16:16
  • Yes BufferedOutputStream doesn't help here. – Stig Jan 14 '17 at 16:19
  • For the sake of futility: have you tried to keep a reference to the RAF and close it to see if the results are different? – bichito Jan 14 '17 at 16:26
  • @efekctive, yes also tried that, same behaviour – Stig Jan 14 '17 at 16:33
  • @Stig then I would try querying the os to figure out who has the handle on the file before you continue. It should save you some wild guessing – bichito Jan 14 '17 at 16:34
  • @efekctive, I think this have been covered in http://stackoverflow.com/questions/2972986/how-to-unmap-a-file-from-memory-mapped-using-filechannel-in-java. Okay the questions is not about deleting the file, but opening the file afterward (same problem) – Stig Jan 14 '17 at 16:44
  • Then you would still need to know who has the file handle – bichito Jan 14 '17 at 16:50
  • 1
    mapping is not very efficient in java if you do not keep the file open for a long time. Using a blocking write in that case is most likely the best option. (And if you fsync at the end it does not matter how fast your write anyway) – eckes Jan 14 '17 at 18:29

0 Answers0