0

I am importing data to neo4j using neoism, and I have some issues importing big data, 1000 nodes, would take 8s. here is a part of the code that imports 100nodes. quite basic code, needs improvement, anyone can help me improve this?

var wg sync.WaitGroup
for _, itemProps := range items {
    wg.Add(1)
    go func(i interface{}) {
        s := time.Now()
        cypher := neoism.CypherQuery{
            Statement: fmt.Sprintf(`
                CREATE (%v)
                SET i = {Props}
                RETURN i
            `, ItemLabel),
            Parameters: neoism.Props{"Props": i},
        }
        if err := database.ExecuteCypherQuery(cypher); err != nil {
            utils.Error(fmt.Sprintf("error ImportItemsNeo4j! %v", err))
            wg.Done()
            return
        }
        utils.Info(fmt.Sprintf("import Item success! took: %v", time.Since(s)))
        wg.Done()
    }(itemProps)
}
wg.Wait()
Hokutosei
  • 2,114
  • 5
  • 24
  • 42

1 Answers1

4

Afaik neoism still uses old APIs, you should use cq instead: https://github.com/go-cq/cq

also you should batch your creates,

i.e. either send multiple statements per request, e.g 100 statements per request

or even better send a list of parameters to a single cypher query:

e.g. {data} is a [{id:1},{id:2},...]

UNWIND {data} as props
CREATE (n:Label) SET n = props
Michael Hunger
  • 41,339
  • 3
  • 57
  • 80
  • hey, thanks for that. changing drivers now would be painful. however, from now on, I will try using the new driver. and thanks for the cypher Tip. I will try that and get back here if I have performance gain. as I am importing big data constantly. – Hokutosei Feb 18 '16 at 09:39
  • thanks! this cypher worked!, was able to import big data like 5k nodes each request, to total of 1M. – Hokutosei Feb 19 '16 at 02:49