0

Currently trying to insert 1.9M records with 8 concurrent threads to local neo4j database, while inserting few hundreds went fine, then I get this exception, though the process is still running, but after going through 1.9M records, only 600k is actually saved, so I was wondering what causes this?

I am using SDN4.1.1.RELEASE and Neo4j 3.0 

org.neo4j.ogm.exception.ConnectionException: Error connecting to remote graph over HTTP
    at org.neo4j.ogm.drivers.http.request.HttpRequest.execute(HttpRequest.java:261)
    at org.neo4j.ogm.drivers.http.request.HttpRequest.executeRequest(HttpRequest.java:204)
    at org.neo4j.ogm.drivers.http.request.HttpRequest.execute(HttpRequest.java:123)
    at org.neo4j.ogm.session.request.RequestExecutor.executeSave(RequestExecutor.java:73)
    at org.neo4j.ogm.session.delegates.SaveDelegate.save(SaveDelegate.java:70)
    at org.neo4j.ogm.session.delegates.SaveDelegate.save(SaveDelegate.java:43)
    at org.neo4j.ogm.session.Neo4jSession.save(Neo4jSession.java:364)
    at org.springframework.data.neo4j.template.Neo4jTemplate.save(Neo4jTemplate.java:203)
    at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
    at com.sun.proxy.$Proxy62.save(Unknown Source)
    at org.springframework.data.neo4j.repository.GraphRepositoryImpl.save(GraphRepositoryImpl.java:48)
    at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.executeMethodOn(RepositoryFactorySupport.java:503)
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:488)
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:460)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:61)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
    at com.sun.proxy.$Proxy76.save(Unknown Source)
    at movies.spring.data.neo4j.controllers.BenchmarkInsertController.saveFacebook(BenchmarkInsertController.java:475)
    at movies.spring.data.neo4j.controllers.BenchmarkInsertController$Neo4jCountDownCallable.call(BenchmarkInsertController.java:549)
    at movies.spring.data.neo4j.controllers.BenchmarkInsertController$Neo4jCountDownCallable.call(BenchmarkInsertController.java:1)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.client.HttpResponseException: Not Found
    at org.neo4j.ogm.drivers.http.request.HttpRequest.execute(HttpRequest.java:238)
    ... 44 common frames omitted

and at line 475:

FBUser neoFacebook = new FBUser();
        neoFacebook.setEmail(usr.getEmail());
        neoFacebook.setId(usr.getFbId());
        neoFacebook.setFbToken(usr.getFbtoken());
        AuthBy auth = new AuthBy();
        auth.setAuth(neoFacebook);
        auth.setUser(neoUser);
        auth.setLastSeen(System.currentTimeMillis());       
        authRepo.save(auth);
Luanne
  • 19,145
  • 1
  • 39
  • 51
kenlz
  • 461
  • 7
  • 22
  • How many cores does your machine have? Do you have enough to run 8 client and 8 server threads concurrently without blocking? – Michael Hunger Jun 17 '16 at 07:09
  • Currently its running on 8 cores with 8 GB of RAM should be sufficient. – kenlz Jun 17 '16 at 07:57
  • @MichaelHunger after reading this post http://stackoverflow.com/questions/9568203/save-method-of-crudrepository-is-very-slow , I think its safe to conclude that It's trying to insert more datas while other insert process hasnt completed. resulting in blocking connection to neo4j? – kenlz Jun 21 '16 at 03:09

1 Answers1

0

Looks like your transactions are timing out. What is the size of the transaction? The recommended size is 1k-10k depending on how many properties you have. Batching your saves in reasonably sized transactions with each transaction using a new OGM session should help. Also if your objects are very deep, you may want to reduce the size of your transaction even further or, split your saves to save related items first and then work up to the parent.

Luanne
  • 19,145
  • 1
  • 39
  • 51
  • Currently I'm inserting 10k at a time with max of 10 properties in 1 label, while the rest of the labels only 3-5 max. I still wonder what is the difference in using repo.save and session.save? I'll try trimming down my batch to 3k and see if it succeeds. – kenlz Jun 17 '16 at 03:37
  • Each FBUser is related to x other entities, so each save of an FBUser will actually save x entities- yes, might want to trim down the number of entities saved. There's no difference between repo.save and session.save. The repositories are a Spring construct and are wrappers around the OGM session saves. – Luanne Jun 17 '16 at 03:38
  • BTW, I would also recommend you upgrade to SDN 4.1.2 (released yesterday) as it contains performance improvements – Luanne Jun 17 '16 at 03:40
  • Will do! I tried 1k, and still too much. It starts to give exception after 100 something inserts, so I guess I'll do 100 at a time. Thanks! – kenlz Jun 17 '16 at 03:46
  • I tried doing 100 and halt the process using Sleep for 10 seconds before doing another 100, and the exception still occurs with SDN 4.1.2. – kenlz Jun 17 '16 at 06:52
  • Are you using a new session for every batch? – Luanne Jun 17 '16 at 07:21
  • The session is autowired, so assuming that it is using the same session. – kenlz Jun 17 '16 at 07:58
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/114918/discussion-between-kenlz-and-luanne). – kenlz Jun 17 '16 at 08:21