5

I am importing 2.3 Billion relationship from a table, The import is not very fast getting a speed on 5Million per hour that will take 20 days to complete the migration. I have heard about the neo4j batch insert and and batch insert utility. The utility do interesting stuff by importing from a csv file but the latest code is some how broken and not running.

I have about 100M relations in neo4j and I have to all check that there shall be no duplicate relationship.

How can I fast the things in neo4j

By current code is like

begin transaction
for 50K relationships
create or get user node for user A
create or get user node for user B
check there is relationship KNOW between A to B if not create the relationhsip
end transaction

I have also read the following:

Community
  • 1
  • 1
maaz
  • 4,371
  • 2
  • 30
  • 48
  • More details about how you currently perform the import would be helpful. The batch services are essentially a way to send multiple commands to the server in a single HTTP request, reducing the HTTP overhead. – Louis-Philippe Huberdeau Dec 11 '12 at 18:54
  • I am reading data from db and importing in embedded neo4j. – maaz Dec 11 '12 at 20:03
  • Do you have some details on what is broken? We would love to fix it. – Peter Neubauer Dec 12 '12 at 12:49
  • creating relationships is slow but not broken and can not find a way to speed things up. – maaz Dec 12 '12 at 12:55
  • 1
    The batch-importer code was fixed (Neo4j API changes), so you can try it again. What do you mean by no duplicate relationship? Can you pre-process your input data for that? E.g. in rels.csv ist should be quite easy to sort by start,end and then remove duplicate lines. – Michael Hunger Dec 16 '12 at 21:11

2 Answers2

3

in case of relationships, and supposing you have enough storage, i would try to not make unique relationships in the import phase - right now i'm actually also importing an SQL table with ~3mil records but i always create a relationship and don't mind whether it is duplicite or not.

you can later after the import simply do a cypher query which will craete unique relationships like this:

START n=node(*) MATCH n-[:KNOW]-m
CREATE UNIQUE n-[:KNOW2]-m;

and

START r=rel(*) where type(r)='KNOW' delete r;

at least this is my approach now and running the later cypher query takes just about minutes. problem could be when you really have bilions of nodes, than the cypher query might fall into an memory error (depends on how much cache you set up for the neo4j engine)

ulkas
  • 5,748
  • 5
  • 33
  • 47
0

How do you do "get user node for user A", lookup from an index? Index lookups really slows batch insertions down. Try to cache as big a part of the users as possible in a simple HashMap "in front" of the index, or use BatchInserterIndex#setCacheCapacity

Mattias Finné
  • 3,034
  • 1
  • 15
  • 7