Bulk load Insert in Elasticsearch with large volume

Question

I have 10GB data, that i need to load into elastic-search index, and I have converted the data to JSON formatted. My question is when i try to load all the data into elastic search using CRUL command its thrown below error. at the same time when I split the JSON files to multiple 1GB files then it works fine. Do we need to follow any methodology to load large size file to elasticsearch or any predefined tools are available ? Please advise here !

Full File(10GB)

curl -XPOST 'http://servername:9200/xyz/tmp/_bulk?pretty' --data-binary @/home/xyz/test.json

Error

curl: (56) Failure when receiving data from the peer

Split File(Success command)

curl -XPOST 'http://servername:9200/xyz/tmp/_bulk?pretty' --data-binary @/home/xyz/test_split1.json
curl -XPOST 'http://servername:9200/xyz/tmp/_bulk?pretty' --data-binary @/home/xyz/test_split2.json

Answer 1

There is a http request size limit of Integer.MAX_VALUE or 2^31-1 which is basically 2GB.

If you check your ES logs you would see something like this HTTP content length exceeded 104857600 bytes and hence you can not index 10GB data at once, you have to split the file.

Please refer to docs. Also this answer would help a lot