1

I want to use Chronicle Queues as inboxes for user messages, with each user of my application having his own queue. However, I'm facing the following "issues":

  1. Since the number of messages per user is not that high, all messages for a single user could be stored in a single queue file without cycling. How can I disable cycling?

  2. In case it is possible to disable cycling, can the whole queue be stored in a single file instead of a directory containing a single queue file plus the directory-listing.cq4t ?

  3. On my Linux OS with EXT4 filesystem, an empty queue uses 83.9Mb of disk space. Can this be reduced to only take roughly the size of the content?

In case one or more of the given issues cannot be circumvented, is there another way to realize user inboxes with Chronicle Queues, like sub-queues or something else?

xpages-noob
  • 1,569
  • 1
  • 10
  • 37
  • An alternative is to have a Chronicle Map with the latest message for any user. The latest message can contain the index of the previous message etc as a linked list. This way you can get all the messages for a user id in linear time. – Peter Lawrey Jan 19 '18 at 11:21
  • @PeterLawrey: Thanks for the hint. As described in a comment below, your solution is similar to the way it is currently implemented. However, due to the "limitation" that I have to set the maximum number of entries of the ChronicleMap in advance, I wanted to check if a queue might be a better solution for storing user mailboxes / message streams. – xpages-noob Jan 19 '18 at 13:28
  • you only need one key per user rather than one per message and Chronicle Map v3 resizes as needed. – Peter Lawrey Jan 19 '18 at 23:09
  • @PeterLawrey: (1.) I misinterpreted your comment: Your suggestion was to have a Chronicle Map for ONLY the latest message of each user in ADDITION to the queue. Of course, that would work too. (2.) If I'm informed correctly, Chronicle Map v3 (I'm using 3.14.5) allows to define a bloating factor that allows to have much more keys than initially set. However, according to Mr. leventov, the performance starts to degrade when the number of entries exceeds the initial number by more than ~20% ( https://stackoverflow.com/a/48059859/1697566 ). – xpages-noob Jan 20 '18 at 09:13
  • on the other hand, map is reasonably performant if it's under utilised. Eg make it 10x what you think it might be. – Peter Lawrey Jan 21 '18 at 07:56

1 Answers1

3

To reduce the size of the files, you can reduce the blockSize used by the queue:

try (final SingleChronicleQueue queue = SingleChronicleQueueBuilder.binary(tmpDir.newFolder()).
    blockSize(4096).
    build()) {

You could store the mailbox name as part of each event:

try (final DocumentContext documentContext = 
    queue.acquireAppender().writingDocument()) {
    documentContext.wire().getValueOut().
    text("user@mailbox").writeText(email);
}

Finally, if you really want to disable the cycle behaviour, you could supply the queue with a clock that never advances:

final AtomicLong fixedClock = new AtomicLong(System.currentTimeMillis());
try (final SingleChronicleQueue queue = SingleChronicleQueueBuilder.binary(tmpDir.newFolder()).
        timeProvider(fixedClock::get).
        blockSize(4096).
        build()) {

Note that this mode of operation is not a supported use-case.

The directory-listing file is required by the internal mechanics of the queue, so there is no way of using the queue without it.

Ensure that you carry out testing when changing any of these configuration parameters.

Mark Price
  • 296
  • 1
  • 3
  • Thanks for your answer and the helpful code snippets. About the second snippet: When I add the mailbox name as an identifier to the event, doesn't that mean that I would have to iterate the whole queue and check each event the get all messages related to a given mailbox? If there are, for example, 100k user with 100 messages each, I would in worst case have to iterate and check 10 million events every time I want to get a single user's mailbox. Or is there some kind of indexing happening within Chronicle queue that I'm not aware of? – xpages-noob Jan 17 '18 at 12:15
  • Yes, you would need to search the whole queue - sorry, your use-case wasn't clear. Without a better idea about what your application requirements are, it's difficult to suggest how to use Chronicle Queue. Given that you have ~100K users with 100 messages, perhaps a database would be a better fit? – Mark Price Jan 18 '18 at 07:58
  • What you suggest is actually the way it is currently implemented: All messages are stored in a ChronicleMap. The keys are generated from the mailbox id plus a continuously increasing index number. This way I can easily get the nth message in each box or iterate through all messages in a box. The main problem is that ChronicleMap is limited to a fixed number of entries. If I allow bloating the performance is said to decrease as soon as there are ~20% more entries than initially set. Hence, I wanted to check if using queues would could an alternative option in my "situation". – xpages-noob Jan 18 '18 at 18:13
  • PS: I know I could replace ChronicleMap with any key-value database, but I did choose it because of its great write performance, solid read performance, and ease of use. – xpages-noob Jan 18 '18 at 18:18
  • Then I think the answer to your initial question is to reduce the blockSize, use a fixed clock, and have a directory per mailbox. You still won't be able to get random access into each mailbox; you'd have to implement something like binary search using the ExcerptTailer indices. It still sounds like a database might be a better fit for your scenario. If you require faster write performance, you could always implement a write-ahead log using a queue, which is persisted to the database in the background. – Mark Price Jan 19 '18 at 10:16
  • Thanks for all your help. Since queues don't really seem to provide a better way to organize the data I'm dealing with, I will leave the currently implemented, ChronicleMap-based solution in place. – xpages-noob Jan 19 '18 at 13:34