1

I have a BlockingCollection and this collection are filling by multi producer functions on different threads.

My consumer function write collection item to a file but isn't scalable beacuse have a single consumer only.

I need multi-producer and multi-consumer implementation.

I can use TextWriter.Synchronized() and lock before Write method but it could waste of same execution time with single TextWriter.

oguzh4n
  • 682
  • 1
  • 10
  • 29

1 Answers1

0

Your textwriter is probably bottlenecked by the IO speed of the disk, it's easy for a mulitocre proc to greatly outperform a HDD. Having multiple writers is not going to remedy that, it's probably going to make it worse because of disk contention.

Assuming you have implemented an efficient wiriter, best solution is to have a maximum number of items in your queue and just accept the IO is the bottleneck here.

If you need to go faster, get faster disk (Solid State or RAID etc), or you could look at binary files and write things to them using a BinaryWriter. Or when you really need the very last drop of speed you can use structs and marshall them. Although this is risky if you want to move your files to other platforms.

Edit: forgot about protobuf.net, that would be my preferred solution to write and read data to disk. It's binary, fast and very easy to use.

Regards Gert-Jan

Community
  • 1
  • 1
gjvdkamp
  • 9,929
  • 3
  • 38
  • 46
  • I already use SSD Disk and I need readable csv files after process so I couldn't use BinaryWriter :( – oguzh4n Mar 06 '12 at 12:34
  • Maybe there's still some stuff that can be optimized if you post the code where you pass turn your data into text. If you post that we could have a look. Why do you need readable CSV? If you have performance issues writing it then it's unlikely to be read by humans. If it needs to be put into some other system there might be faster ways to do that.. – gjvdkamp Mar 06 '12 at 12:59
  • @oguzh4n _"I already use SSD Disk"_ That means multiple threads won't degrade performance, but they still won't help. Use as many I/O threads as you have (independent) disks. – H H Mar 12 '12 at 23:20