0

I need to persist some number very frequently - more than 1000 times a second.

I have tried :

  • Properties.store that wraps FileOutputStream
  • direct use of FileOutputStream
  • Files.write
  • RandomAccessFile

In all cases, I have kept the file descriptor open and just spilled data inside.

The results are an average of 500 actions per second (... and I need more than 1000 actions per second, my optimal goal is 10,000 actions per second)

What other options do I have?

Michael
  • 2,835
  • 2
  • 9
  • 15
  • 3
    What level of assurance do you require that the data has been written to disk / persistent storage? This is key to working out the best solution. – Stephen C Oct 06 '19 at 06:16
  • 1
    One thing to consider is the physics. With a spinning disc, you can only write or rewrite a given disk block once every disc rotation. So for example, a 7200 rpm disk can rewrite a physical disk block *at most* 120 times a second. To get a "commit" rate faster than that, you would need a non-volatile cache in front of the hard disk(s). – Stephen C Oct 06 '19 at 06:24
  • 2
    You should store the changes in memory, and only commit the changes to disk once every second or so. – tonayy Oct 06 '19 at 09:01
  • @user7153055 what's the amount of data per operation, in bytes? En voge are append only approaches ("immutable database") would that be an option, maximizes throughput? Given that info one could calculate more numbers. – Curiosa Globunznik Oct 07 '19 at 13:25
  • The good ol' RAM disk, with every ten coffee-minutes a file backup.. – Joop Eggen Oct 07 '19 at 15:00

0 Answers0