Define elasticsearch mapping for a dictionary coming from the remote api

Question

I am fetching marketcatalogue data from betfair api-ng and I want to store it in to elasticsearch (v1.4.4).

The data coming from API contains lot of properties and complex types. There is a complex type named runners which contains related data and a Dictionary<string,string>. I want to define such mapping so it will store data in Elasticsearch. Sample mapping is below:

"marketcatalogue" :{
    "properties":{
        "marketId":{"type":"string", "index": "not_analyzed" },
        "marketName":{"type":"string", "analyzer":"keylower" },
        "isMarketDataDelayed":{"type","boolean"},
        "description":{
            "persistenceEnabled":{"type","boolean"},
            "bspMarket":{"type","boolean"},
            "marketTime":{"type" : "date","format":"dateOptionalTime"},
            "suspendTime":{"type" : "date","format":"dateOptionalTime"},
            "settleTime":{"type" : "date","format":"dateOptionalTime"},
            "bettingType":{"type":"integer"},
            "turnInPlayEnabled":{"type","boolean"},
            "marketType":{"type":"string", "analyzer":"keylower" },
            "regulator":{"type":"string", "analyzer":"keylower" },
            "marketBaseRate":{"type":"double"},
            "discountAllowed":{"type","boolean"},
            "wallet":{"type":"string", "analyzer":"keylower"},
            "rules":{"type":"string"},
            "rulesHasDate":{"type","boolean"},
            "clarifications":{"type":"string"}
        },
        "runners":{
            "selectionId":{"type":"long"},
            "runnerName":{"type":"string", "analyzer":"keylower"},
            "handicap":{"type":"double"},
            "metadata":{

            }
        }
    }
}

}

The medatadata is a Dictionary<string,string> coming from the API and it can contain data like:

<"TRAINER_NAME", "John">, <"WEARING", "Wearing one">,....

Storing the data into type is not a big deal but the problem is how to define the mapping for the dictionary.

Any help will save lot of my time and will make me to learn the mapping creation the better way.

Thanks in advance.

score 4 · Accepted Answer · answered Jun 06 '16 at 22:46

Every single field in Elasticsearch can be both a singular value, or an array of values. This includes objects:

"metadata" : {
  "type" : "object",
  "properties" : {
    "key" : { "type" : "string", "index" : "not_analyzed" },
    "value" : { "type" : "string", "index" : "not_analyzed" }
  }
}

Then if your JSON looked like this, it would be picked up with the above mapping:

{
  ...,
  "metadata": [
    { "key" : "TRAINER_HOME", "value" : "John" },
    { "key" : "WEARING", "value" : "Wearing one" }
  ]
}

Alternatively, if you don't even want to search the data, but you want it to be accepted into the index as part of the _source, but not actually index the individual fields (this reduces the size of the index if you're not actually going to use the metadata for search):

{
  "metadata" : {
    "type" : "object",
    "dynamic" : false
  }
}

This means you can never do anything with that data inside of ES, but it will be there when you query against other fields.

If you do actually want to search the metadata, then you probably want to use "type" : "nested" rather than object (be sure to take the note at the bottom seriously; nothing is free).

Define elasticsearch mapping for a dictionary coming from the remote api

1 Answers1