0

I too am beginning to look at ChronicleQueue.

From the documentation:

Reading the queue follows the same pattern as writing, except there is a possibility there is not a message when you attempt to read it

I am trying to understand what happens when there is no message. The beauty of a LinkedBlockingQueue is that it just works, with little CPU load, when one thread is putting items into a queue (say, from a stream) and another thread is consuming items (yes, removing elements as they arrive and processing them in small groups). I would like to take advantage of the mapped files for history for raw items from the stream and the performance of a ChronicleQueue, but it seems that to replicate the behavior of a LinkedBlockingQueue my code will have to keep track of excerpt indices to know what excerpts have been processed - either when they arrive (placed in queue), or through some machinations using replay in the code that actually process the content of an excerpt. That said, I suspect I am missing something fundamental in my understanding of appender/tailer behavior. Any insights would be appreciated.

Jack Copper
  • 33
  • 1
  • 9

1 Answers1

0

Chronicle Queue is designed for low-latency, single-threaded environments and hence it never blocks on any operations.

Therefore, the usual pattern of reading from the queue involves busy-spinning (or various types of pausers, the simplest being Thread.yield()), e.g.:

while (true) {
    try (DocumentContext dc = tailer.readingDocument()) {
        // false here means there's no data to read, so we just keep waiting
        if (dc.isPresent()) {
             Bytes bytes = dc.wire().bytes();
            // read from bytes
        }
    }
}

Unless you want to return to certain items in your queue later, you don't need to bother with indices or anything of this sort.

NB: busy-spinning is opposed to blocking and it WILL consume 100% of CPU, and that's a deliberate trade-off for low-latency solutions.

Dmitry Pisklov
  • 1,196
  • 1
  • 6
  • 16
  • thanks for the clarification. I have been able to successfully add stream events to separate queues. Now the problem is reading the queues to synchronize and combine events. It seems tailers cannot be interleaved. I will put details in a new question. – Jack Copper Jan 08 '22 at 15:41
  • I suggest you use the SAME queue for multiple events, that will be consumed by same consumer. In the meantime, could you mark this response as accepted? – Dmitry Pisklov Jan 10 '22 at 12:30
  • The problem with using the same queue is that the events are different types of events. I suppose it might be possible to figure out how to read/write at the lowest level, but I think that would be a nightmare (for me or anyone else in the future) making sure they get the right number of bytes in the right order (i.e., reading/writing objects is much more straightforward). – Jack Copper Jan 10 '22 at 14:59
  • @JackCopper you need to research the MethodReader/MethodWriter. They make reading/writing different types of messages really easy, and that's the Chronicle's recommended way – Dmitry Pisklov Jan 11 '22 at 09:11