go json.Unmarshal to interface some clause disappeared

Question

i have a sentence to query a group of users by terms (keyword field present uid) and with a range limit (long filed present unixtime ),the sentence can be executed in Kibana and curl, but when I use golang client(https://github.com/olivere/elastic) to perform the query, after json. Unmarshal(), the sentence is tampered,the range request is abandoned, why? my sentence disobey the json's rule?

package main

import (
    "encoding/json"
    "fmt"
)

var hot_cache map[string]byte
var followers []string

var prefix = "{\"constant_score\" : {\"filter\" : {\"bool\" : {\"filter\" : {\"range\" : {\"unixtime\" : {\"gte\" : %d, \"lte\" : %d}}}, \"filter\" : {\"terms\" : {\"uid\" : ["
var suffix = "]}}}}}}"

func main() {

    tmp := prefix
    tmp += "\""
    tmp += "123"
    tmp += "\""
    tmp += suffix

    qstr := fmt.Sprintf(tmp, 1, 2)
    fmt.Println("raw: ", qstr)

    var f interface{}
    err := json.Unmarshal([]byte(qstr), &f)
    if err != nil {
        panic(err)
    }

    fmt.Println("json: ", f)
}

Output:

 raw:  {"constant_score" : {"filter" : {"bool" : {"filter" : {"range" : {"unixtimestamp" : {"gte" : 1, "lte" : 2}}}, "filter" : {"terms" : {"uid" : ["123"]}}}}}}

json: map[constant_score:map[filter:map[bool:map[filter:map[terms:map[uid:[123]]]]]]]

any one knows why?

Your raw json is broken: Your "bool" object contains two fields of the same name "filter". Just pretty print your qstr and you'll see. Protip: If it doesn't work you botched up the input. — Volker, Nov 21 '17 at 08:24
thanks a lot,but this sentence can be execute in Kinaba or curl with elastic server, so this my be elastic server's problem? — user8095277, Nov 21 '17 at 08:31
If elasticsearch decides to handle the broken JSON in a specific way it might be a curtesy of ES or a backward compatibility hack or whatnot. Your JSON still has two fields of the same name and unmarshaling that will drop one. There is absolutely nothing you can do about it but fix the JSON. — Volker, Nov 21 '17 at 08:58
You want to use two filters under bool, have the filter value be an array of objects just like in the example here https://www.elastic.co/guide/en/elasticsearch/reference/current/query-filter-context.html — mkopriva, Nov 21 '17 at 09:00

maerics · Answer 1 · 2017-11-21T14:36:41.143

Your "qstr" input JSON string contains the key constant_score.filter.bool.filter twice, which causes the parser to retain the second value for it.

{
  "constant_score": {
    "filter": {
      "bool": {
        "filter": { "range": { "unixtime": { "gte": 1, "lte": 2 } } },
        "filter": { "terms": { "uid": [123] } }
      }
    }
  }
}

JSON syntax does not disallow repeated keys; however, it doesn't require that parsers handle them in any special way either, so the vast majority of JSON parsers choose to store the last value of a repeated key in an object, and go's unmarshaler is no different.

Here's an example - consider the following valid JSON string:

{"a":1,"a":2}

Suppose you deserialized this document into an object (e.g. map, dictionary, k/v store, etc). Since keys are unique in such data structures, what value should be stored for they key "a", 1 or 2?

// JavaScript
JSON.parse('{"a":1,"a":2}'); // => {a: 2}

# Ruby
JSON.parse('{"a":1,"a":2}') # => {"a" => 2}

# Python
json.loads('{"a":1,"a":2}') # => {u'a': 2}

// Go
var obj interface{}
err := json.Unmarshal([]byte(`{"a":1,"a":2}`), &obj)
if err != nil {
  panic(err)
}
fmt.Printf("%#v\n", obj)
// map[string]interface {}{"a":2}

go json.Unmarshal to interface some clause disappeared

1 Answers1