5

Is there a maximum length for the key and/or value ByteIterable in Xodus? If there is a hard limit, what is that limit (i.e. how many bytes)? And what will happen if a ByteIterable exceeds that limit?

Alex
  • 1,141
  • 1
  • 13
  • 24
Martin Häusler
  • 6,544
  • 8
  • 39
  • 66
  • Well, one obvious upper bound is that the `length` is of type `int` - and it seems to be implemented using arrays, so the usual limits for arrays push it down a few bytes more. – Hulk Nov 09 '18 at 08:35
  • Good point. But that is the *theoretical* limit. I assume that the underlying on-disk tree will have its own limits? – Martin Häusler Nov 09 '18 at 08:38
  • Probably, but I don't know anything about that, just briefly glanced at the SourceCode of the interface on GitHub – Hulk Nov 09 '18 at 08:41
  • As `ByteIterable` is an interface and there are multiple imlementations, I'm not sure there can be a general answer. – Hulk Nov 09 '18 at 08:43
  • I was hoping that one of the JetBrains folks who develop this library would find this issue and may provide an answer. Usually, libraries like this have a "feel good zone" where everything runs smooth, but once the keys in particular become too big, things slow down (but don't crash). The question is not as trivial as it seems, hence I would like to see a statement from the devs. – Martin Häusler Nov 09 '18 at 08:56
  • It seems that these things are not really designed to work with large numbers of bytes - for example, [CompoundByteIterable](https://github.com/JetBrains/xodus/blob/master/openAPI/src/main/java/jetbrains/exodus/CompoundByteIterable.java) does not check wheter the sum of lengths overflows int - and I'm not sure what will happen in this case in the code using such values. – Hulk Nov 09 '18 at 09:40

1 Answers1

5

Xodus is a log-structured database, all changes are written sequentially to a log, which is an infinite sequence of .xd files. On the lowest level, any key/value pair is written as a single record which can only exist in a single .xd file. That means that the sum of sizes of key and value can't exceed the size of single .xd file. Even more, it's not good to store key/value pairs of size close to the size of single .xd file since this would create a database with much unmovable (by means of the database garbage collector) free space. An attempt to write too big key/value pair would fail with TooBigLoggableException thrown.

The size of single .xd file is controlled by the EnvironmentConfig.LOG_FILE_SIZE setting. It's 8388608 bytes (0x800000, 8MiB) by default, so the answer to the question is like the maximum length of key and value is 8388608 (0x800000) bytes.

To deal with data of arbitrary size, use Virtual File Systems.

Vyacheslav Lukianov
  • 1,913
  • 8
  • 12