elasticsearch search request size limit error

Question

me elasticsearch version 7.9.3 (running on ubuntu) holds an index of each day (logs) so when a query needs to include for example data from 2020-01-01 until 2020-11-20

Search query will look like this: (which returns error 400)

http://localhost:9200/log_2020-02-14,log_2020-02-26,log_2020-02-27,log_2020-04-24,log_2020-04-25,log_2020-07-17,log_2020-08-01,log_2020-09-09,log_2020-09-21,log_2020-10-06,log_2020-10-07,log_2020-10-08,log_2020-10-16,log_2020-10-17,log_2020-10-18,log_2020-10-21,log_2020-10-22,log_2020-11-12/_search?pretty

I know I can split the request into two but I don't see why (4096 bytes over HTTP it's not so big)

any chance to config this issue ?

response:

{
    "error": {
        "root_cause": [
            {
                "type": "too_long_frame_exception",
                "reason": "An HTTP line is larger than 4096 bytes."
            }
        ],
        "type": "too_long_frame_exception",
        "reason": "An HTTP line is larger than 4096 bytes."
    },
    "status": 400
}

Val · Answer 1 · 2020-11-20T07:02:18.563

1

URLs cannot exceed a certain size depending on the medium. Elasticsearch limits that length to 4096 bytes.

Since you seem to be willing to query all indexes of 2020 since January 1st until today (Nov 20), you can use a wildcard like this:

http://localhost:9200/log_2020*/_search?pretty

Another way is by leveraging aliases and put all your 2020 indexes behind the log_2020 alias:

POST /_aliases
{
  "actions" : [
    { "add" : { "index" : "log_2020*", "alias" : "log_2020" } }
  ]
}

After running that you can query the alias directly

http://localhost:9200/log_2020/_search?pretty

If you want to make sure that all your daily indexes get the alias upon creation you can add an index template

PUT _index_template/my-logs
{
  "index_patterns" : ["log_2020*"],
  "template": {
    "aliases" : {
        "log_2020" : {}
    }
  }
}

UPDATE

If you need to query between 2020-03-04 and 2020-09-21, you can query the log_2020 alias with a range query on your date field

POST log_2020/_search
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2020-03-04",
        "lt": "2020-09-22"
      }
    }
  }
}

edited Nov 20 '20 at 07:02

answered Nov 20 '20 at 06:28

Val

207,596
13
358
360

thanks that help but let's say i want all the days between 2020-03-04 and 2020-09-21 how do I alias this ? – David Munsa Nov 20 '20 at 06:43
I guess your indexes also have a date field in it, right? if that's the case you can query the whole alias but add a `range` query on that date field with the desired min/max dates – Val Nov 20 '20 at 07:00
It wont affect profrmanc?, Its not much faster to query only the index u actually need? – David Munsa Nov 20 '20 at 13:04
That's the whole point of running queries over aliases. ES is smart is enough to figure out which indexes to search. Try it out and you'll see – Val Nov 20 '20 at 13:04

elasticsearch search request size limit error

1 Answers1