1

I was using neo4j 3.1.0 enterprise edition. The main logical in my graph is: There are "IP" nodes and "User" nodes and both have "UNIQUE" constraints. Each time an user login, I add a relationship from IP to User.

Here is my insert Cypher:

MERGE (i:IP {ip:"1.2.3.4"}) 
MERGE (u:User {username:"xxx@gmail.com"}) 
MERGE (i) - [l:SUCC] -> (u) 
SET i:ExpireNode, i.expire={expire} 
SET u:ExpireNode, u.expire={expire}
SET l.expire={expire}, l.login={login}

The insert is pretty fast as the beginning. But when the number of node grows to millions, it became very slow and sometimes took more than 1 second to insert nodes and relationships.

How could I optimize it? I was running neo4j with 12-cores CPU and 64G memory. The initial head size is 16G and page cache is 30G.

--------------------------------------------------------------

Tested the same cypher in Web UI and it took 10ms for commands. But by using java driver, it will sometimes take more than 1s. Below is my java code:

try (Transaction tx = session.beginTransaction()) {
    for (Login login : loginList) {
        Value value = login2Operation(login);
        tx.run(INSERT_COMMANDS_SUCC, value);
    }
    tx.success();
}

--------------------------------------------------------------

After some exploring, I found that the insert speed increased significantly if running in 5 threads. But the overall speed is too slow and I have to increase to 100 threads. Then the single insert of each speed grows to 1s. So, I believe the problem is because the parallel ability of Neo4j.

In the neo4j.conf, I added dbms.threads.worker_count=200. But it's not helping. Any ideas?

Weizhou He
  • 202
  • 2
  • 12
  • 1
    Can you PROFILE a run of this query and paste in the results (after expanding all elements of the profile)? – InverseFalcon Feb 28 '17 at 11:19
  • @InverseFalcon Good Point! I tried the command in Neo4j web UI, and it only takes 10ms. However, it seems took more than 1s when using neo4j-java-driver. I guess the problem is because the java-driver? Added the java code for more information – Weizhou He Mar 02 '17 at 08:47
  • 1
    You may want to review [Michael Hunger's tips and tricks](http://jexp.de/blog/2017/03/5-tips-tricks-for-fast-batched-updates-of-graph-structures-with-neo4j-and-cypher/#_unwind_to_the_rescue) for batches and insertions. Instead of running your operations in a for loop, you should probably collect all relevant parameters into a list, then make a single run call, and in your run UNWIND the collection and apply your changes all at once. – InverseFalcon Mar 02 '17 at 11:04

1 Answers1

1

Thanks to @InverseFalcon 's advices and the UNWIND operation helps a lot!

Get more details in Michael Hunger's tips and tricks

Weizhou He
  • 202
  • 2
  • 12