elasticsearch / kibana errors "Data too large, data for [@timestamp] would be larger than limit

Question

On my test ELK cluster, I'm encountering the following error when trying to see data from the last week.

Data too large, data for [@timestamp] would be larger than limit

The warning about shards failing appears to be misleading because the elasticsearch monitoring tools kopf and head show that all shards are working properly, and the elastic cluster is green.

enter image description here

One user in the google group for elasticsearch suggested increasing ram. I've increased my 3 nodes to 8GB each with a 4.7GB heap, but the issue continues. I'm generating about 5GB to 25GB of data per day, with a 30 day retention.

spuder · Answer 1 · 2019-03-14T16:28:55.160

59

Clearing the cache alleviates the symptoms for now.

http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-clearcache.html

Clear a single index

curl -XPOST 'http://localhost:9200/twitter/_cache/clear'

Clear multiple indicies

curl -XPOST 'http://localhost:9200/kimchy,elasticsearch/_cache/clear'

curl -XPOST 'http://localhost:9200/_cache/clear'

Or as suggested by a user in IRC. This one seems to work the best.

curl -XPOST 'http://localhost:9200/_cache/clear' -d '{ "fielddata": "true" }'

Update: these errors went away as soon as the cluster was moved to a faster hypervisor

edited Mar 14 '19 at 16:28

answered Apr 22 '15 at 23:05

spuder

17,437
19
87
153

I ran into the same issues and clearing the cache solved them. I'm not sure if this is something which needs to be done periodically. Thanks! – Carlos Apr 30 '15 at 08:44
This worked! Thanks! Never seen this issue before despite using ES for several years now...very strange. – Joseph Siefers Jun 30 '15 at 16:35
Yes, had the same problem and cleared the cache and everything was working again. Upvote and star question. – Gman Dec 07 '15 at 17:13
Only the last one worked for me, even though I was only querying one of the indices – Jonas Bergström Jan 19 '16 at 12:01
@spuder +1, It helped a lot. – user60679 Feb 23 '16 at 11:42
Are you doing this regularly or was it enough to do it once? – selle Mar 02 '16 at 09:55
But the question remains, does clearing the cache have to be done periodically or there is a permanent fix such as configuration change that needs to be done to fix it permanently? – smishra May 20 '16 at 23:27
Thanks @spuder it works..But is it necessary to clear cache to periodically? Why cache becomes full...is there any option to avoid this problem – Ranjeet Sep 23 '16 at 05:58
The problem magically went away for us. We suspect network/hypervisor performance problems were the root cause. – spuder Sep 23 '16 at 13:44
also needs `-H "Content-Type: application/json"` – Hendry Feb 04 '20 at 21:22
"[POST /_cache/clear] does not support having a body" – OrangeDog Nov 02 '21 at 09:46

score 10 · Answer 2 · answered Sep 06 '19 at 13:28

10

The problem is the memory given to the Elasticsearch by ES_JAVA_OPTS.

Try to give more memory with: ES_JAVA_OPTS="-Xmx2g -Xms2g".

answered Sep 06 '19 at 13:28

FChiri

707
8
14

score 5 · Answer 3 · answered Jun 27 '20 at 01:21

The _cache API also supports targeted clearing to the threadpools for fielddata, request, etc. through just normal URI query parameters.

Prototype, see the placeholder <cache type>:

$ curl -XPOST \
    -H "Content-Type: application/json" \
    'http://localhost:9200/_cache/clear?<cache type>=true'

Examples

$ curl -XPOST \
    -H "Content-Type: application/json" \
    'http://localhost:9200/_cache/clear?fielddata=true'

$ curl -XPOST \
    -H "Content-Type: application/json" \
    'http://localhost:9200/_cache/clear?request=true'

NOTE: You can also target specific indexes, substitute below where <index>:

$ curl -XPOST \
    -H "Content-Type: application/json" \
    'http://localhost:9200/<index>/_cache/clear?request=true'

References

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-clearcache.html

The Content-Type is not necessary if you're not including a body — OrangeDog, Nov 02 '21 at 09:51

score 1 · Answer 4 · answered Sep 05 '18 at 20:06

Clearing the cache didn't work for our cluster. When checking the various nodes using http://x.x.x.x:9200/_cat/indices?v&s=index:desc one was giving the above error and the others had invalid pointer errors. I restarted the elastic service on the one giving the rate limit/data too large error. When it came back online there were some unassigned shards, which I fixed by dropping the replication count to a lower number (only need to do this on one of the nodes to change the index settings for the cluster):

IFS=$'\n'
for line in $(curl -s 'elastic-search.example.com:9200/_cat/shards' | fgrep UNASSIGNED); do
  INDEX=$(echo $line | (awk '{print $1}'))
  echo start $INDEX
  curl -XPUT "elastic-search.example.com:9200/$INDEX/_settings" -d '{
      "index" : {
        "number_of_replicas" : 1
      }
    }
    '
done

# Check shard/cluster status (may take some time to rebalance):
# http://elastic-search.example.com:9200/_cat/shards?v&s=index:desc
# http://elastic-search.example.com:9200/_cluster/health?pretty

Also https://discuss.elastic.co/t/data-too-large/32141 seems to mention it may be an issue with the size of the JVM heap.

elasticsearch / kibana errors "Data too large, data for [@timestamp] would be larger than limit

4 Answers4

Examples

References

Linked