2


I get a fast stream of data (objects) and I would like to write it to a file.
This is a stand alone process so it doesn't do anything but read the data from a socket parse it to csv and write all to a file.

What is the best way to write a lot of csv lines to a file?
Is a buffer writing my solution?
Is there a buffered File object in Java ?
Should I manage it myself and use writeLines()?

Bick
  • 17,833
  • 52
  • 146
  • 251
  • 1
    What, now? It needs to be very fast, but not very efficient? – Lukas Eder May 23 '11 at 15:51
  • It means that I dont do any computation on the data . So I only need it to be fast enough in order to write all. I will fix my question. – Bick May 23 '11 at 15:54

3 Answers3

9

Fastest way to write huge data in text file Java

Community
  • 1
  • 1
bpgergo
  • 15,669
  • 5
  • 44
  • 68
3

If you're dealing with a huge throughput of data then I suggest you use a set of in-memory buffers where you deposit the data arriving and then have a thread/threadpool which uses Java NIO to "consume" these buffers and write them onto disk. You will however be limited by the disk writing speed -- bear in mind that it's not unusual for the speed of network to be faster than the speed of your hard disk! so you might want to consider a threadpool which writes in different physical locations and only "pastes" these files after all the data has been received and written.

Liv
  • 6,006
  • 1
  • 22
  • 29
1

As mentioned above, chances are that its disk I/O that limits you, not Java abstractions.

But beyond using a good lib to deal with CSV, you might consider using other (even more) efficient formats like JSON; as well as compression. GZIP is good at compressing things, but relatively slow; but there are faster ones too. For example, LZF (like this Java implementation) is fast enough to compress at speeds higher than typical disk I/O (and uncompress even faster). So compressing output may well increase throughput as well as reduce disk usage.

StaxMan
  • 113,358
  • 34
  • 211
  • 239