Spring Data Neo4j memory consumption on "supernode" entities

Question

As far as I know, once a NodeEntity in Spring Data Neo4j is loaded, the default behaviour is to lazily load its relations by fetching only ids of related nodes. While it seems quite ok in most situation, I have doubts about it in the case of so called "supernodes" - the nodes that have numerous relations to other nodes. That kind of nodes, even if small by themselves, will hold a huge collection of ids, using more memory than we would like it to use, and possibly being not "lazily loaded enough" in effect...

So my question is - how shall I deal with that kind of supernode?

My first idea is to simply remove all @RelatedTo/@RelatedToVia mappings (or at least the ones with relation types that are "numerous") from that kind of nodes and simply bypass SDN when operations on those relations are needed, and use SDN in other cases. Does it seem to have sense? Do you have some other suggestions or some experience in that kind of situations?

score 2 · Answer 1 · edited May 23 '17 at 11:51

2

I have not worked with SDN but I will give a try to the approximation of metanodes. With this approximation you build a structure that split the total number of relations into the number of metanodes (if a node has 1000 connections and you use 10 metanodes, each metanode will have 100 connection while the supernode just 4. You can see a graphic representation in the folowing image: https://i.stack.imgur.com/DMQGs.png.

In this way you can have a good control of how many relations can have a node and therefore how many node will be maximal loaded by SDN.

You can read more about it on http://neo4j.com/book-learning-neo4j/ and also in this similar post Neo4j how to avoid supernodes

edited May 23 '17 at 11:51

Community

1
1

answered Aug 07 '15 at 11:44

alacambra

549
5
12

Thanks for your answer. I'm more or less familiar with a methods of getting rid on supernodes. But my question is not about getting rid of supernodes, but rather about getting rid of memory impact that supernodes (intentionally placed in DB) may have on SDN. – cichystefan Aug 07 '15 at 11:51
Maybe I have not understand exactly the problem, but so as I know, the reduction of the number of the connected nodes will result in less nodes loaded. Have I missed the point? – alacambra Aug 07 '15 at 12:01

score 1 · Answer 2 · answered Aug 07 '15 at 16:35

First, can you provide the version of SDN you are using so we can target the question to the right maintainers of the library.

Secondly, while I don't know really the internals of SDN but have worked heavily with other OGMs, my understanding of LazyLoading is quite different that the one you provide, for the simple reason that lazy loading the ids can be very harmful in the sense that you can have corrupted data if another process is deleting one of the nodes having one of these ids.

Generally, and it is quite common in other OGMs, in the case of an object has no annotations representing relationships, you would just recreate the object from his metadata and the loaded node.

However if it has relationships, you would then create a proxy of that object that will extend the entity itself.

The entity values on the proxy will not be instantiated in the first instance, you would then override all getters and add in the proxy the methods for retrieving the related nodes (so the Entity manager would be injected in the proxy).

So basically, a proxy will be empty until you call one of the getters on it.

You can also "fine-grain" this behavior by creating Custom repositories that extend the default one, in the sense you can choose to only LAZY_LOAD one type of relationships and EAGER_LOAD the others.

The method described by albert makes lot of sense in some cases, however it is hard to accomplish on the basic OGM side, you would better have a BehaviorComponent that will handle this for you during lifecycle events, or add some kind of pagination to the getter method, which I think is not part of the OGM right now.

score 1 · Accepted Answer · answered Aug 08 '15 at 17:56

For supernodes I'd just not specify the relationship on the supernode entity. But only on the related nodes.

And if you're interested in the relationship you either lookup the related node and follow to the supernode.

Or if you really need to load the millions of relationships, use a cypher statement.

You can also put the many relationships on a separate node for that purpose or add a tree-like-substructure which also allows to deal with subselections.

Spring Data Neo4j memory consumption on "supernode" entities

3 Answers3