0

I am designing a REST API which works according to the asynchronous design detailed here. I am using RabbitMQ to enqueue the initial requests - so the client makes a call, receives a 202 Accepted response, and the job is enqueued by the server. In order that clients can get status updates ('percent done') on tasks we have a secondary queue resource, just as in the linked article.

Given that each task has its own queue resource it seems we need one temporary RabbitMQ queue per task. I am wondering whether this is a wise design choice, even though I can't really see any other options. It seems unlikely to be very efficient, and I am uneasy about the possibility of having lots of temporary queues being created like this, especially as I cannot see a way to guarantee that they will all be cleaned up (despite RabbitMQ's auto-delete facility). Prior to RabbitMQ I was using SQS for this, and have painful experience of what can happen in this respect.

I note that a similar type of queue management will be already familiar to those using RabbitMQ in RPC style. Is there a possible alternative, however?

ChrisM
  • 2,128
  • 1
  • 23
  • 41

1 Answers1

5

Firs of all, each queue utilize apr. 20k memory, so having a lot of them is up to you and your hardware. But in general, it smells. Really.

For status updates I see nothing wrong to use some key-value database, like redis or even memcache and update percent done there. Thus status check (as well as updating) will be damn fast, simple and lightweight.

Update:

I can suggest further architecture:

  1. Client POST task payload to some endpoint, say /tasks.
  2. Application generate unique task id (uuid aka guid is your friend here), published that task with it id to RabbitMQ queue and then return id to client.
  3. Workers (one or many) consume tasks from RabbitMQ and depends of processing step update Redis key which has task id with some value (step, percentage done, estimated time to receive result). So, it may be looks like SET task:{id} "<some valye>". When task completed by worker it MAY update Redis key with task result or store it somewhere else and then set Redis key represent task is finished.
  4. Client MAY time to time GET /tasks/{id} to receive task status or it result.
  5. When Application receive GET /tasks/{id} it return task status represented by Redis key (GET task:{id}). If key is not set (nil) then task is not yet taken by worker.

P.S.

RPC is something different from what you asked, but i would recommend to read this question for some details.

Community
  • 1
  • 1
pinepain
  • 12,453
  • 3
  • 60
  • 65
  • @zach178miami Indeed, it smells. Something lightweight is definitely required. I had heard of Redis being used for status updates in this type of scenario, but it's not clear to me how I might actually implement that. I guess it would be a case of Redis queues, with one queue per job? Or updating a single entry by overwriting it? – ChrisM Feb 11 '14 at 21:45
  • 1
    Redis MAY be a good solution, it depends of messages flow, say, 10k msg/sec x 1-2kb is not what Redis queues stands for, I guess. Also pay note that RabbitMQ can dump messages on disc (persistent queues + persistent messages) so you will not lose any of tasks. – pinepain Feb 12 '14 at 07:15
  • 1
    Upd my answer. Can you also specify language you are using to implement what you want? – pinepain Feb 12 '14 at 07:29
  • Thanks, very helpful. Our API is written in Clojure, and our backend workers are in Java, with some Clojure. We use the langohr Clojure RabbitMQ client. In fact the architecture we have in mind (and have had until recently, though with SQS) is virtually identical to what you outlined. It's very unlikely we'll be at 10k msg/sec any time soon, and in any case the messages (both input and status updates) are tiny. We store our actual data in S3, and the messages simply point to it. I guess updating/overwriting Redis keys (rather than using queues) would be feasible in that case. – ChrisM Feb 12 '14 at 07:52
  • 1
    Actually it makes far more sense to just update a key than to use a Redis queue. Because if a task takes 2 mins, and if for some reason the client only comes back after 10, they will surely just want the latest status update (in this case 100%, which means the server will return `303 See Other`, so they get redirected to the created resource). They won't want to have to loop through all the previous updates, which are irrelevant by that point anyway. – ChrisM Feb 12 '14 at 18:28