57

I want to see all queries executed against an elasticsearch instance. Is it possible to run elasticsearch in a debug mode, or to tell it to store all queries executed against it?

The purpose is to see which queries are launched from a software using elasticsearch for analysis.

Chris Stryczynski
  • 30,145
  • 48
  • 175
  • 286
paweloque
  • 18,466
  • 26
  • 80
  • 136

4 Answers4

30

In versions of ElasticSearch prior to 5, you can accomplish this by changing the ElasticSearch.yml configuration file. At the very bottom of this file, you can adjust the logging time to record all:

index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms

index.search.slowlog.threshold.fetch.warn: 1s  
index.search.slowlog.threshold.fetch.info: 800ms
index.search.slowlog.threshold.fetch.debug: 500ms
index.search.slowlog.threshold.fetch.trace: 200ms

index.indexing.slowlog.threshold.index.warn: 10s
index.indexing.slowlog.threshold.index.info: 5s
index.indexing.slowlog.threshold.index.debug: 2s
index.indexing.slowlog.threshold.index.trace: 500ms

Adjust the settings and restart your node, then consulting the logs to view the queries executed against your node. Note if in production log files will rapidly increase in size.

James
  • 1,394
  • 2
  • 21
  • 31
Nathan Smith
  • 8,271
  • 3
  • 27
  • 44
22

In version 5.x, you have to set slow log logging per index.

Command line:

curl -XPUT 'http://localhost:9200/myindexname/_settings' -d '{
"index.indexing.slowlog.threshold.index.debug" : "0s",
"index.search.slowlog.threshold.fetch.debug" : "0s",
"index.search.slowlog.threshold.query.debug" : "0s"
}'

Or, if you are using Kibana, go to the Dev Tools bar and enter:

PUT /myindexname/_settings 
{"index.indexing.slowlog.threshold.index.debug": "0s", 
"index.search.slowlog.threshold.fetch.debug" : "0s", 
"index.search.slowlog.threshold.query.debug": "0s"}

#1: Apply to ALL indices

You can apply the setting to ALL indices with the following command:

PUT /_all/_settings 
{"index.indexing.slowlog.threshold.index.debug": "0s", 
"index.search.slowlog.threshold.fetch.debug" : "0s", 
"index.search.slowlog.threshold.query.debug": "0s"}

#2: Preserve existing settings

If you don't want to overwrite existing settings, but just add new ones, add '''preserve_existing=true''' after _settings, like this:

PUT /_all/_settings?preserve_existing=true 
{"index.indexing.slowlog.threshold.index.debug": "0s", 
"index.search.slowlog.threshold.fetch.debug" : "0s", 
"index.search.slowlog.threshold.query.debug": "0s"}

The above request will ONLY add the settings if they don't exist. It will not change them if they are already there.

#3: All available log settings

All available slow log settings are here and below for your reference:

PUT /test_index/_settings
{
"index.search.slowlog.threshold.query.warn": "60s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.query.debug": "1s",
"index.search.slowlog.threshold.query.trace": "0.1s",
"index.search.slowlog.threshold.fetch.warn": "30s",
"index.search.slowlog.threshold.fetch.info": "5s",
"index.search.slowlog.threshold.fetch.debug": "1s",
"index.search.slowlog.threshold.fetch.trace": "0.1s",
"index.indexing.slowlog.threshold.index.warn": "6s",
"index.indexing.slowlog.threshold.index.info": "5s",
"index.indexing.slowlog.threshold.index.debug": "1s",
"index.indexing.slowlog.threshold.index.trace": "0.1s",
"index.indexing.slowlog.level": "info",
"index.indexing.slowlog.source": "1000"
}
IvanD
  • 7,971
  • 4
  • 35
  • 33
  • 2
    In a single line so its convenient next time I'm back (in a month): curl -XPUT 'http://localhost:9200/_all/_settings' -d '{"index.indexing.slowlog.threshold.index.debug": "0s", "index.search.slowlog.threshold.fetch.debug" : "0s", "index.search.slowlog.threshold.query.debug": "0s"}' – kwerle Jun 28 '19 at 20:46
  • Note: the output now appears in a logfile including the name "slowlog". For me it was `elastic_5_5_index_search_slowlog.log` – Otheus Oct 18 '19 at 09:02
15

Starting with Version 5 ElasticSearch charges money for this functionality. It's called "Audit log" and is now part of X-Pack. There is a basic license available that is free, but this license only gives you a simplistic monitoring functionality. Authentication, query logging and all these rather basic things cost money now.

Toumal
  • 562
  • 4
  • 10
1

Yes, it's possible to tell Elasticsearch to log all queries executed against it and you can configure logging levels, such as DEBUG. You can change it in ES 7.13.x using curl:

curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "transient": {
    "logger.org.elasticsearch.discovery": "DEBUG"
  }
}
'

On macOS log files are stored on $ES_HOME by default. Please check the docs about Elasticsearch Logging

Ricardo
  • 3,696
  • 5
  • 36
  • 50
  • I feel like this is the correct answer, but I am not so sure `logger.org.elasticsearch.discovery` is the correct Logging Hierarchy to catch queries. Do you know what are the available Logging Hierarchies? – Elouan Keryell-Even Mar 08 '22 at 15:52
  • 1
    You're right, the give example gives logs related to the [`discovery`](https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-discovery-hosts-providers.html) process. I don't know on top of my head the package for the search queries. – Ricardo Mar 08 '22 at 19:03
  • 1
    @ElouanKeryell-Even, can you try something like `logger.org.elasticsearch.search`? Based on the docs, the log hierarchy seems to mimic the package name. See the [search package](https://github.com/elastic/elasticsearch/tree/010493682c94ee4f4384a3b2bba85294bd5b8f9e/src/main/java/org/elasticsearch/search). Please let me know if it works so I can update the answer! – Ricardo Mar 08 '22 at 19:10