Elasticsearch mapping - different data types in same field

Question

I am trying to to create a mapping that will allow me to have a document looking like this:

{
    "created_at" : "2014-11-13T07:51:17+0000",
    "updated_at" : "2014-11-14T12:31:17+0000",
    "account_id" : 42,
    "attributes" : [
    {
        "name" : "firstname",
        "value" : "Morten",
        "field_type" : "string"
    },
    {
        "name" : "lastname",
        "value" : "Hauberg",
        "field_type" : "string"
    },
    {
        "name" : "dob",
        "value" : "1987-02-17T00:00:00+0000",
        "field_type" : "datetime"
    }
]

}

And the attributes array must be of type nested, and dynamic, so i can add more objects to the array and index it by the field_type value.

Is this even possible?

I have been looking at the dynamic_templates. Can i use that?

score 26 · Answer 1 · answered Nov 30 '18 at 15:59

You actually can index multiple datatypes into the same field using a multi-field mapping and the ignore_malformed parameter, if you are willing to query the specific field type if you want to do type specific queries (like comparisons).

This will allow elasticsearch to populate the fields that are pertinent for each input, and ignore the others. It also means you don’t need to do anything in your indexing code to deal with the different types.

For example, for a field called user_input that you want to be able to do date or integer range queries over if that is what the user has entered, or a regular text search if the user has entered a string, you could do something like the following:

PUT multiple_datatypes
{
  "mappings": {
    "_doc": {
      "properties": {
        "user_input": {
          "type": "text",
          "fields": {
            "numeric": {
              "type": "double",
              "ignore_malformed": true
            },
            "date": {
              "type": "date",
              "ignore_malformed": true
            }
          }
        }
      }
    }
  }
}

We can then add a few documents with different user inputs:

PUT multiple_datatypes/_doc/1
{
  "user_input": "hello"
}

PUT multiple_datatypes/_doc/2
{
  "user_input": "2017-02-12"
}

PUT multiple_datatypes/_doc/3
{
  "user_input": 5
}

And when you search for these, and have ranges and other type-specific queries work as expected:

// Returns only document 2
GET multiple_datatypes/_search
{
  "query": {
    "range": {
      "user_input.date": {
        "gte": "2017-01-01"
      }
    }
  }
}

// Returns only document 3
GET multiple_datatypes/_search
{
  "query": {
    "range": {
      "user_input.numeric": {
        "lte": 9
      }
    }
  }
}

// Returns only document 1
GET multiple_datatypes/_search
{
  "query": {
    "term": {
      "user_input": {
        "value": "hello"
      }
    }
  }
}

I wrote about this as a blog post here

This is helpful when indexing single values but it won't work with multi values where at least one of them is not convertible. Example: `PUT multiple_datatypes/_doc/2 { "user_input": ["hello", "2017-02-12"] }`. Even setting `ignore_malformed=true` at index level does not help here. I think this is an issue in elasticsearch. — Fabrizio Fortino, Jun 05 '23 at 09:36

score 4 · Accepted Answer · answered Nov 14 '14 at 14:47

4

No - you cannot have different datatypes for the same field within the same type.

e.g. the field index/type/value can not be both a string and a date.

A dynamic template can be used to set the datatype and analyzer based on the format of the field name

For example: set all fields with field names ending in "_dt" to type datetime.

But this won't help in your scenario, once the datatype is set you can't change it.

answered Nov 14 '14 at 14:47

Olly Cruickshank

6,120
3
33
30

1

Okay. Crap. I must rethink me idea :) Thanks – Morten Hauberg Nov 14 '14 at 15:06
1

Just as an addition, do not try to find the solution in the same field and a different type. That also does not work. – Jettro Coenradie Nov 14 '14 at 18:59
The emphasis I need from @JettroCoenradie post, **different type**; and presumably it's because the same field is shared across different mapping types [as seen here](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#field-conflicts) – Nate Anderson Apr 24 '16 at 18:45
This answer is either out of date, incorrect, or incomplete. The docs do provide a solution. You can update and reindex: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#_updating_existing_field_mappings – Malik A. Rumi Nov 28 '17 at 22:19

Elasticsearch mapping - different data types in same field

2 Answers2