71

I have ElasticSearch index created which has approx 350 fields (including nested fields), I have defined mapping only for few of them. While calling the _update API I am getting below exception with 400 Bad request.

Limit of total fields [1000] in index [test_index] has been exceeded

I want to understand the exact reason for this exception since my number of fields and fields for which mapping is defined both are less than 1000.

Note: I have nested fields and dynamic mapping enabled.

bfontaine
  • 18,169
  • 13
  • 73
  • 107
SSG
  • 1,265
  • 2
  • 17
  • 29

5 Answers5

98

You can fix the issue by increasing the value of index.mapping.total_fields.limit (default 1000)

index.mapping.total_fields.limit

The maximum number of fields in an index. Field and object mappings, as well as field aliases count towards this limit. The default value is 1000.

To increase total fields limit to 2000, try this

PUT test_index/_settings
{
  "index.mapping.total_fields.limit": 2000
}

The reason to limit the number of fields is :

Defining too many fields in an index is a condition that can lead to a mapping explosion, which can cause out of memory errors and difficult situations to recover from. This is quite common with dynamic mappings. Every time a document contains new fields, those will end up in the index’s mappings

Related Resources :
1. Get the number of fields on an index
2. Settings to prevent mappings explosion

Ashraful Islam
  • 12,470
  • 3
  • 32
  • 53
  • Thanks for the quick reply ! Actually I checked the number of fields in my index using the link you shared , its 928 , so I am not able to figure out the exact cause – SSG Mar 27 '19 at 10:02
  • 3
    @SSG Maybe you are trying to index new document that contains more than 72 new fields – Ashraful Islam Mar 27 '19 at 10:43
12

I landed here via google. This worked for me (from the shell):

curl -s -XPUT http://localhost:9200/test-index/_settings  -H 'Content-Type: application/json' -d '{"index.mapping.total_fields.limit": 2000}'
ThomasArdal
  • 4,999
  • 4
  • 33
  • 73
Marco
  • 3,470
  • 4
  • 23
  • 35
  • 1
    If somebody decides just to copy/paste this.. There should be empty space between -d option and header content. – Lvka Jan 05 '22 at 11:28
  • I downvoted because this doesn’t answer the question, in addition to duplicate a previous answer. – bfontaine Mar 27 '23 at 08:21
5

Another approach to handle this is to carefully design the mappings as the OP seems to be doing and to turn off dynamic mapping by setting dynamic = false (or even dynamic = strict)

Note that this approach can be applied to the entire mapping or to other properties/nested objects within, which enables quite a bit good degree of flexibility.

Reference to Dynamic mapping in the docs

elachell
  • 2,527
  • 1
  • 26
  • 25
0

Another possible solution, esp. when working in Groovy context: check that your data is a (well-formatted) String instance. In my case, I had been transforming my mapped data to Json using myMappedData as JSON in Groovy. The solution was to use (myMappedData as JSON).toString(). See https://stackoverflow.com/a/71938642/4420271.

philburns
  • 310
  • 4
  • 18
0
PUT project-documents/_settings
{
  "index.mapping.total_fields.limit": 1100
}

Use this with caution. Do not increase the size too much as it will degrade the performance.

Jojo Joseph Kunathucherry
  • 46
  • 3