Sorted, concurrent queue implementation (twisted) with sqlite?

Question

I want to design an application using the twisted framework with some deferreds adding multi-field elements to a queue.

The queue is consumed by the application with other deferreds. The problem is I want to consume the queue elements not by order of addition but give higher priority to some elements based on the element field content.

My current (probably faulty) idea was to have a generator which executes a sql select which produces a results set with in the proper priority and iterate over this result to generate deferreds. The problem is, while the generator is iterating over the result set further elements may have been added to the queue by deferreds.

Is there a way to do this without executing a new select every time the generator is called? I.e. can the "result set" and the iterator cursor be automatically be "updated"?

If not, how would you implement this?

Answer 1

The Queue module has a PriorityQueue class which may meet your needs.

Answer 2

Why not use a Python list, using the module "heapq" from the stdlib to keep it in priority order? (Your list elements would be tuples with (priority, time_of_insertion, objects) - and since tthe data is kept in normal Python list, you can pass it around with no issues within a twisted app.

I think it would be easier than using sqlite if all that you need is sort by priority.

(In this answer I place an example usage of heapq that could help: heapq with custom compare predicate )