16

I will be getting a JSON string every .01 seconds or even faster and I have to insert it in some collection so that I can later loop through every row and do some processing and delete that row after that. The process of insertion keeps on going. I am a bit confused as whether to use ArrayList or Queue.

So basically I would insert the first message then second and then ... and simultaneously another process would read the first inserted value for processing and deleting that record followed by second inserted value. Please advise which one would be fastest and takes less storage? Or any other collection would best suit my requirement?

Edit: I have two methods -

GetMessages() - which keeps on getting/adding messages in queue (here the messages are received from only one source which keeps on sending messages and does not stop till we manually stop the process)

ProcessMessages() - which will read the queue records and after processing delete them from the queue

Dmitry Bychenko
  • 180,369
  • 20
  • 160
  • 215
user1254053
  • 755
  • 3
  • 19
  • 55
  • 13
    `ArrayList`? That hasn't been recommended for at least 10+ years. You should only ever be using anything from `System.Collections.Generic`. – user9993 Jun 14 '17 at 09:00
  • 2
    The length of the strings is entirely irrelevant as the collections only store references. The number of string references is the only relevant thing and “every .01 seconds” isn’t mindblowingly fast… – Holger Jun 14 '17 at 11:06

3 Answers3

30

Sounds like you should use a ConcurrentQueue<string>, where one thread can push values that another thread could pop.

Definitely don't use or even think of using the pre-generic ArrayList type ever again.

You'd still need some bookkeeping code that checks whether your processing is running faster than the insertion, otherwise you're guaranteed to run out of memory at some point.

CodeCaster
  • 147,647
  • 23
  • 218
  • 272
  • 1
    I often see new devs using the rubbish from `System.Collections`. I guess their Java background makes them think that all collection types from Java have a 1:1 mapping to .NET's collection types. – user9993 Jun 14 '17 at 09:01
  • 1
    I am reading about the ConcurrentQueue and the example shows the use of for loop to iterate in the queue. But mine queue will keep on increasing or decreasing so how would I know till when to loop? I think I can iterate with - if (!concurrentQueue.IsEmpty) , correct? – user1254053 Jun 14 '17 at 10:58
  • 1
    If you want to use it like that, [Dmitry's answer](https://stackoverflow.com/a/44539335/266143) is a better one. – CodeCaster Jun 14 '17 at 11:08
  • I am getting the messages from a server so there is only one producer (with respect to blockingcollection). I just want to ensure that the messages received in the order should be processed in the same order. I am reading on both the options. Thank you guys for letting me know on these options. – user1254053 Jun 14 '17 at 11:16
  • 1
    @user1254053 why iterate the queue with a for / for each loop at all? Just check if the queue is non-empty, take the head and do something with it. – bash0r Jun 14 '17 at 12:21
21

It seems that you are looking for a Producer-Consumer design pattern (Wikipedia Producer–consumer_problem).

  1. Producer creates Json's
  2. Consumer processes them

If it's your case, try BlockingCollection which is specially designed for this.

Patrick Hofman
  • 153,850
  • 22
  • 249
  • 325
Dmitry Bychenko
  • 180,369
  • 20
  • 160
  • 215
  • 4
    This may indeed be an even better type for this problem, see also https://stackoverflow.com/questions/10010748/what-are-the-differences-between-concurrentqueue-and-blockingcollection-in-net – CodeCaster Jun 14 '17 at 08:25
  • @Dmitry Bychenko - thanks for the suggestion. Is this works on FIFO? – user1254053 Jun 14 '17 at 10:32
  • @user1254053: FIFO can be ensured, see https://stackoverflow.com/questions/3825275/does-blockingcollectiont-guarantee-removal-order – Dmitry Bychenko Jun 14 '17 at 10:39
11

From how I read your question you have basically two requirements:

  • It is a queue;
  • It is safe to concurrently queue and dequeue.

I would use ConcurrentQueue: it gives the queuing mechanism you are talking about and it is also thread-safe, so you can add and remove at the same time without the need to lock the queue yourself.

Which one is best performance and memory wise is hard to tell without having exact information. If you have a lot of repeating strings, you could try to intern them or create a cache yourself where you keep repeating elements.

Mathias R. Jessen
  • 157,619
  • 12
  • 148
  • 206
Patrick Hofman
  • 153,850
  • 22
  • 249
  • 325
  • I prefer @Dmitry Bychenko's answer. This is clearly a typical producer-consumer pattern, and a `BlockingCollection` is a much better idea. It already wraps a `ConcurrentQueue` internally anyway. – user9993 Jun 14 '17 at 09:02