Index 400 billion documents in solr 6.3

Question

I have to index about 400 billion documents to solr 6.3. I am using pysolr to parse my csv data before indexing. How I can speedup my indexing. In order to index a document to solr, it used add method that has following syntax bydefault

add(self, docs, boost=None, fieldUpdates=None, commit=True, softCommit=False, commitWithin=None, waitFlush=None, waitSearcher=None, overwrite=None, handler='update')

One basis option, is that I should make commit and softcommit to false for fast indexing. Is it right way?

Any other option to peroform fast indexing?

You can get good advice in http://stackoverflow.com/questions/12328969/speed-up-solr-indexing — AR1, Jan 25 '17 at 09:37
Any reason why you can't submit your CSV directly in batches? The CSV handler with batches is usually quite a fast combination. — MatsLindh, Jan 26 '17 at 09:47
I do not know how to submit CSV directly in batches. Pleae share some details — Hafiz Muhammad Shafiq, Jan 27 '17 at 04:33

score 1 · Answer 1 · answered Jan 29 '17 at 06:12

See if you commit in single go it will be very memory expensive. So better option is to commit in batches so what I would suggest is to keep the count variable

if(count == 10000)
{
perform solr commit operation
}

Also, make your indexing script multi-threaded to fastly complete these batches.

Index 400 billion documents in solr 6.3

1 Answers1