42

I read notes about Lucene being limited to 2Gb documents. Are there any additional limitations on the size of documents that can be indexed in Elasticsearch?

Rashack
  • 4,667
  • 2
  • 26
  • 35
Asimov4
  • 2,776
  • 3
  • 23
  • 38

2 Answers2

63

Lucene uses a byte buffer internally that uses 32bit integers for addressing. By definition this limits the size of the documents. So 2GB is max in theory.

In ElasticSearch:

There is a max http request size in the ES GitHub code, and it is set against Integer.MAX_VALUE or 2^31-1. So, basically, 2GB is the maximum document size for bulk indexing over HTTP. And also to add to it, ES does not process an HTTP request until it completes.

Good Practices:

  • Do not use a very large java heap if you can help it: set it only as large as is necessary (ideally no more than half of the machine’s RAM) to hold the overall maximum working set size for your usage of Elasticsearch. This leaves the remaining (hopefully sizable) RAM for the OS to manage for IO caching.
  • In client side, always use the bulk api, which indexes multiple documents in one request, and experiment with the right number of documents to send with each bulk request. The optimal size depends on many factors, but try to err in the direction of too few rather than too many documents. Use concurrent bulk requests with client-side threads or separate asynchronous requests.

For further study refer to these links:

  1. Performance considerations for elasticsearch indexing

  2. Document maximum size for bulk indexing over HTTP

slm
  • 15,396
  • 12
  • 109
  • 124
Utsav Dawn
  • 7,896
  • 2
  • 29
  • 44
  • 1
    For maximum HTTP request size, see : https://github.com/elastic/elasticsearch/blob/148265bd164cd5a614cd020fb480d5974f523d81/docs/reference/modules/http.asciidoc – vvs14 Mar 16 '16 at 11:41
  • What happens in case of compression? Does it limit based on compressed or raw size? – Marko Švaljek Apr 16 '19 at 09:52
  • Google promptly answers ["elasticsearch maximum document size"](https://www.google.com/search?q=elasticsearch+maximum+document+size) with 100 kB. This is based on [this summary](https://www.elastic.co/guide/en/app-search/current/limits.html), which contains the hint "configurable". How does this answer relate to that configuration? Do you assume the value has been set to its maximum? – bluenote10 Jan 29 '21 at 18:07
  • @bluenote10 the documentation page you mention refers to App Search, which is a product built on top of Elasticsearch – mcont Jan 19 '23 at 13:14
11

Think things have changed slightly over the years with Elasticsearch. In the 7.x documentation referenced here - General Recommendations:

Given that the default http.max_content_length is set to 100MB, Elasticsearch will refuse to index any document that is larger than that. You might decide to increase that particular setting, but Lucene still has a limit of about 2GB.

So it would seem that ES has a limit of ~100MB and Lucene's is 2GB as the other answer stated.

slm
  • 15,396
  • 12
  • 109
  • 124