2

I have not found how to save a general list of primitive types, e.g. ints, or strings, in a property of an entity. I might have missed something obvious...

https://github.com/JetBrains/xodus/wiki/Entity-Stores described that "only Java primitive types, Strings, and ComparableSet values can be used by default".

It seems not hard to convert an Iterable into a ComparableSet. However it is a Set.

I will take a look into PersistentEntityStore.registerCustomPropertyType() to see if that helps. I just feel wrong to do that to just save a list of integers.

Links seemed to be able to serve as a way of saving a list of Entitys. But it seems there is no addProperty() counterpart to addLink().

Appreciated if some one can share a way or a workaround for this, or maybe why this is not supported.

Thanks

Vin
  • 559
  • 2
  • 18
  • 1
    Allowing duplicates in a list is a must, right? – Vyacheslav Lukianov Dec 10 '20 at 14:53
  • Yes. What I am looking for is a general `List`-like structure. So both duplicates and order. – Vin Dec 10 '20 at 16:15
  • 1
    It seems there is no direct ways to persist a list. The entity links as I tested are acting like a `Set`, with no duplicates allowed. What I came up with was to implement a `ComparableList` and a `ComparableListBinding`, mostly adapted from the built-in `ComparableSet` and `ComparableSetBinding`, then register them with `.registerCustomPropertyType()`. – Vin Dec 13 '20 at 08:01
  • 1
    @VyacheslavLukianov I found https://youtrack.jetbrains.com/issue/XD-355. I don't seem to be able to see comments in the issue, so I have no idea what happened. But https://github.com/JetBrains/xodus/commit/680c14f suggested `ComparableSet` resulted from it. Why a `Set` was used instead of a `List`? I have not read a lot, but from `ComparableSet` and `ComparableSetBinding`, it seemed only `getMinimum()`, `getMaximum()` and `minus()` would benefit from a `Set`. However I think those should not be hard to come by with a `List`, while ordering and duplicates are banned in a `Set`. Your thoughts? – Vin Dec 13 '20 at 15:58
  • 1
    Entity properties allow searching for entities by a property value or in a range of values. ComparableSet allows to define a property with several values, single property cannot have several equal values, and all values are naturally sorted (as Comparables). That's why a set, not a list. Do you need to search for entites by a value in a list? – Vyacheslav Lukianov Dec 13 '20 at 17:04
  • Yes, it'd be great as what's being done with `ComparableSet`. I did notice that `ComparableSet` got some special treatment when indexing, e.g. https://github.com/JetBrains/xodus/blob/e54b983/entity-store/src/main/java/jetbrains/exodus/entitystore/tables/PropertiesTable.java#L174. However, the underlying key-value store does do sorting and de-duplicating anyway. So for indexing, a set doesn't seem required for functionality. For serialization, `ComparableSetBinding`'s `writeObject()` and `readObject()` don't seem to need a set particularly. My observations so far, may miss things I don't know. – Vin Dec 14 '20 at 01:49
  • 2
    I'd suggest then a hybrid scheme. Serialize a list to binary stream or string and save it using `setBlob(..)` or `setBlobString(..)`. For searching, create entities of another type having single property which is equal to a value in a list. Those entities would be like an enum or a dictionary of values. Use `addLink(..)` and `deleteLink(..)` to maintain links to dictionary items. Source entities (having a value in their lists) could be found using `StoreTransaction.findLinks(String, Entity, String)` method. – Vyacheslav Lukianov Dec 14 '20 at 09:52
  • @Vin can you share your solution? – quarks Dec 24 '20 at 05:29
  • @quarks I posted one of my attempts on the issue below. Hopefully it might be of some use. – Vin Dec 31 '20 at 10:10

1 Answers1

1

As mentioned in the comments, one workaround I came up with was to create a ComparableList, by adopting code from ComparableSet.

The idea was to make a list that is able to convert to and from an ArrayByteIterable, and register it with .registerCustomPropertyType(). To do that, 2 classes are needed, ComparableList and ComparableListBinding. I'm sharing one iteration I used at the bottom. By the way, I made them immutable, comparing to the mutable ComparableSet. The newly implemented types should be registered once in a transaction of the store before using them.

That should allow you to store and retrieve a list. However the items in ComparableList would not get indexed as they would when saved in a ComparableSet -- there are some special treatment for ComparableSet in the entity store implementation. So without modifying the library, indexing would work only with hacks like creating another property to just index the values.

I was considering to implement a different entity store that could better support lists, on top of the xodus key-value store, bypassing the xodus entity store entirely. That might be a better solution to the list issue we are talking about here.


ComparableList:

@SuppressWarnings("unchecked")
public class ComparableList<T extends Comparable<T>> implements Comparable<ComparableList<T>>,
    Iterable<T> {
  @Nonnull
  private final ImmutableList<T> list;

  public ComparableList(@Nonnull final Iterable<T> iterable) {
    list = ImmutableList.copyOf(iterable);
  }

  @Override
  public int compareTo(@Nonnull final ComparableList<T> other) {
    final Iterator<T> thisIt = list.iterator();
    final Iterator<T> otherIt = other.list.iterator();
    while (thisIt.hasNext() && otherIt.hasNext()) {
      final int cmp = thisIt.next().compareTo(otherIt.next());
      if (cmp != 0) {
        return cmp;
      }
    }
    if (thisIt.hasNext()) {
      return 1;
    }
    if (otherIt.hasNext()) {
      return -1;
    }
    return 0;
  }

  @NotNull
  @Override
  public Iterator<T> iterator() {
    return list.iterator();
  }

  @Nullable
  public Class<T> getItemClass() {
    final Iterator<T> it = list.iterator();
    return it.hasNext() ? (Class<T>) it.next().getClass() : null;
  }

  @Override
  public String toString() {
    return list.toString();
  }
}

ComparableListBinding:

@SuppressWarnings({"unchecked", "rawtypes"})
public class ComparableListBinding extends ComparableBinding {

  public static final ComparableListBinding INSTANCE = new ComparableListBinding();

  private ComparableListBinding() {}

  @Override
  public ComparableList readObject(@NotNull final ByteArrayInputStream stream) {
    final int valueTypeId = stream.read() ^ 0x80;
    final ComparableBinding itemBinding = ComparableValueType.getPredefinedBinding(valueTypeId);
    final ImmutableList.Builder<Comparable> builder = ImmutableList.builder();
    while (stream.available() > 0) {
      builder.add(itemBinding.readObject(stream));
    }
    return new ComparableList(builder.build());
  }

  @Override
  public void writeObject(@NotNull final LightOutputStream output,
      @NotNull final Comparable object) {
    final ComparableList<? extends Comparable> list = (ComparableList) object;
    final Class itemClass = list.getItemClass();
    if (itemClass == null) {
      throw new ExodusException("Attempt to write empty ComparableList");
    }
    final ComparableValueType type = ComparableValueType.getPredefinedType(itemClass);
    output.writeByte(type.getTypeId());
    final ComparableBinding itemBinding = type.getBinding();
    list.forEach(o -> itemBinding.writeObject(output, o));
  }

  /**
   * De-serializes {@linkplain ByteIterable} entry to a {@code ComparableList} value.
   *
   * @param entry {@linkplain ByteIterable} instance
   * @return de-serialized value
   */
  public static ComparableList entryToComparableList(@NotNull final ByteIterable entry) {
    return (ComparableList) INSTANCE.entryToObject(entry);
  }

  /**
   * Serializes {@code ComparableList} value to the {@linkplain ArrayByteIterable} entry.
   *
   * @param object value to serialize
   * @return {@linkplain ArrayByteIterable} entry
   */
  public static ArrayByteIterable comparableSetToEntry(@NotNull final ComparableList object) {
    return INSTANCE.objectToEntry(object);
  }
}
Vin
  • 559
  • 2
  • 18
  • Hi, you mean you were able to make saving work but `find`ing based on what's in the List you were not able to make it work? I mean when you said *So without modifying the library, indexing would work only with hacks like creating another property to just index the values.* – quarks Dec 31 '20 at 11:38
  • Same case with this: https://youtrack.jetbrains.com/issue/XD-833 – quarks Dec 31 '20 at 11:40
  • This is about `find()`. But XD-833 seems to be about a different issue? -- `ComparableSet` could not serialize a custom `Comparable` type. The line I said was about https://github.com/JetBrains/xodus/blob/e54b983/entity-store/src/main/java/jetbrains/exodus/entitystore/tables/PropertiesTable.java#L174, which is how the index is created, and is called when saving a property. In that, `String`s are lower cased before indexing, and `ComparableSet` iterates each item for indexing, while other primitive types are indexed directly. – Vin Dec 31 '20 at 12:17
  • I realized I made a typo, which was probably the cause of our confusions here. Sorry for that! Copy-pasta failed. I just struck out the error and corrected in the paragraph containing the line you referenced. – Vin Dec 31 '20 at 12:26
  • XD-833 is about serializing a custom type and also be able to query it later – quarks Jan 01 '21 at 05:28