0

I am designing solution for multi-threaded Producer Consumer problem.

product are very much in numbers (around 100,000) because consumer runs once in an Hour.

there are multiple producer threads and one consumer thread.

Consumer thread is a Task that executes every hour.

Consumer thread saves data to database for the last hour, for the current hour, it leaves it in the product collection.

I begin writing solution as described in following post classic producer consumer pattern using blockingcollection and tasks .net 4 TPL

I am thinking more optimized way of producing and consuming products, for this i thought not to check for last hour in each and every row of product collection, instead use ConcurrentDictionary with key as Hour(DateTime).

Thus instead of using Product I want to use something like ConcurrentDictionary>

I think by using ConcurrentDictionary on Hour(DateTime) Key, Consumer can easily pick last hour Products and save it to database without having burden of filtering products by hour.

I thought to use BlockingCollection in following ways but both seems not correct.

Using BlockingCollection>> does not seems good as inside it there will be only one entry that itself is a dictionary.

Using ConcurrentDictionary>> does not seems to give advantages of BlockingCollection collection as it is not directly accessed by the consumer/producer.

How can I use BlockingCollection in this scenario? Am I on the right approach?

EDIT: This server takes very high load i.e. receives 4 MB of data every 30 seconds and process it that took 50 time more of memory. Its a server that is expected to run for months.

Community
  • 1
  • 1
Imran Rizvi
  • 7,331
  • 11
  • 57
  • 101
  • Why does the consumer not process the current hour? And why do you run the consumer only once and hour? – paparazzo Jun 13 '13 at 18:12
  • @blam see the following question , it has detail about he server http://stackoverflow.com/questions/17084421/how-to-know-debug-profile-hanged-c-sharp-application-in-the-given-situation?noredirect=1#comment24722679_17084421 – Imran Rizvi Jun 13 '13 at 18:36
  • So the server has a high load. How does the consumer running once an hour address load? A filter only add more load. I get it is not efficient to process 1 at a time. But why not process 1000 at a time. – paparazzo Jun 13 '13 at 18:57
  • @blam Server insert received data to database, database is the actual bottleneck, it takes time to insert data and thus memory load increases on server, this Product is actually an startification of data.However you are right we can process 1000 at a time, but what it has to do with BlockingCollection question? – Imran Rizvi Jun 13 '13 at 21:34
  • Everything. You are not on the right approach. – paparazzo Jun 13 '13 at 22:37
  • @Blam can you explain? – Imran Rizvi Jun 13 '13 at 22:43
  • Could, but that link is no way an answer to my question. I parse data and load into SQL at a rate of 30,000 rows and 30 mb a minute using a BlockingCollection. – paparazzo Jun 13 '13 at 23:16
  • @Blam Consumer not process the current hour Because I want to save data every hour to database. I run it once an hour because I want to consume once in an hour. – Imran Rizvi Jul 05 '13 at 14:58

0 Answers0