0

I receive a big amount of data to my web api service that takes a while to process (Parse, process data, put in DB, etc).

The problem I'm having is that the processing is not always finished when the next chunk of data comes. I don't want to process the data simultaneously but rather wait for the first call to finish before processing the next.

I've looked into RabbitMQ and other solutions similar - the problem is that I don't have a process running checking the queue at all times "while(true)" style.

Any suggestions ?

hallizh
  • 150
  • 1
  • 10
  • 1
    You say you don't have a process running that can consume messages from the queue as they arrive, but it sounds like this is exactly what you need. Is there a reason why you can't create a windows service or similar for this purpose? – Ben Robinson Nov 12 '14 at 10:37
  • The web api is pushing results after processing each document to the client via SignalR. – hallizh Nov 12 '14 at 10:40

1 Answers1

2

You can't handle a queue without a running process. If you make your API call wait untill the processing slot is ready you'll get timeouts.

You must use a solution like RabbitMQ and a worker process that process incoming data. Nothing prevents this process from pushing result using SignalR.

Guillaume
  • 12,824
  • 3
  • 40
  • 48
  • Say I'd respond instantly with 200 an then just throw up a thread (I don't need to respond to the request, I can just always respond with "OK") Anyway, how would I create this processing slot in the Web API? – hallizh Nov 12 '14 at 10:51
  • 1
    http://stackoverflow.com/questions/536681/can-i-use-threads-to-carry-out-long-running-jobs-on-iis – Guillaume Nov 12 '14 at 10:59
  • Thank you, this is what I meant. But I'm starting to realize that it's a bad idea :) – hallizh Nov 12 '14 at 11:16