3

The ChronicleQueueBuilder interface (4.5) allows parameterisation of the queue file characteristics. By default I get 80mb files for daily rollover.

  • Is there any guidance how one should use these values?
  • What are the trade-offs? Can I optimize it to let an appender seek to an index quickly?
  • If I know my average excerpt size and the average number of excerpts per period, can I choose these values to avoid resizing of the queue file during the day?
MartyIX
  • 27,828
  • 29
  • 136
  • 207
vasquez
  • 583
  • 5
  • 18

1 Answers1

3

Unless you have a reason to change it I would leave it as it is. If you make the chunk size smaller it might use less space on windows though on linux it uses sparse files so it won't make a difference.

You could increase the chunk size to 1 GB or more however if you are concerned about the cost of resizing I suggest calling the pretoucher() in another thread periodically to ensure there is always headroom.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • How do I change the blockSize from default of max 16MB to a value higher than that in the code ? Any help is appreciated – Anirudh Kashyap Dec 18 '18 at 05:48
  • @AnirudhKashyap The default is 64MB, you can call `ChronicleQueueBuilder.blockSize(128 << 20)` to change to 128 MB for example. – Peter Lawrey Dec 18 '18 at 09:08
  • May be I am confused with block size and chunk size. I see this on FAQ page. "I want to store large messages; what is the limit? The limit is about 1 GB, as of Chronicle 4.x. The practical limit without tuning the configuration is about 16 MB. At this point you get significant inefficiencies, unless you increase the data allocation chunk size." What is this 16 MB that we are referring to ? – Anirudh Kashyap Dec 18 '18 at 19:52
  • Also, is there any performance impact if I change that to 1 GB, which I believe is the max that could be set ? – Anirudh Kashyap Dec 18 '18 at 22:15
  • 1
    @AnirudhKashyap in the latest versions you can set the size to many GB. The main downside is; you can't use 4GB or more on windows, but you can on Linux and how much apparent space you waste, though on Linux memory and disk are allocated lazily. – Peter Lawrey Dec 19 '18 at 08:57
  • Thanks Peter. Can you also shed some light on chunk size and block size and let's say I specify my block size to be 100MB, I read that the max size for write should be not greater than 25 MB. Why is this the case ? – Anirudh Kashyap Dec 19 '18 at 20:48
  • @AnirudhKashyap The chunks are allocated with overlapping regions. This overlap is 25% of the chunk size. If you start writing near the end of a chunk, you can only write 25% in one go. With lots of smaller writes you can write more an a chunk, but 25% is always safe. – Peter Lawrey Dec 19 '18 at 20:56
  • Also, in the case where I can have varied sized messages being written, what is the best block size that needs to be set ? set it to max ? – Anirudh Kashyap Dec 19 '18 at 21:01
  • This will be my last question. In my above example of 100MB block size and 25MB overlap size, will there be a 100 MB block allocated off heap and we keep writing chunks of max of 25 MB in that block ? – Anirudh Kashyap Dec 20 '18 at 00:09
  • The first block allocated is 125% of the chunk size. e.g. the default is 64 MB but the file starts with 80 MB. Every time it grows by 64 MB. – Peter Lawrey Dec 20 '18 at 08:29
  • Thanks a lot Peter, that helps. – Anirudh Kashyap Dec 20 '18 at 19:13