5

I want to remove data from elasticsearch with configuring the data date period, for example, data older that 30 days, or maybe by size, like if index size is bigger than 100Mb then remove old data. I use logstash to move logs to one index in elasticsearch. How can i do that?

SolarCore
  • 133
  • 2
  • 16

2 Answers2

7

If you want to explicitly delete logs based on ur criteria you can use delete_by_query

POST /my_logs/_delete_by_query
{
    "query": {
        "range": {
            "date": {
                "lte": <your_target_date>
            }
        }
    }
}
Kaushik J
  • 962
  • 7
  • 17
  • Thank you, but first will try the solutions that leandrojmp and tanimak provided – SolarCore Jun 24 '20 at 15:30
  • curl example: curl http://localhost:9200/customer/_delete_by_query -X POST -d '{"query":{"range":{"date":{"lte":"2021/08/13"}}}}' -H "Accept: application/json" -H "Content-Type: application/json" Be weary that "lte" stands for less-than-equals – Chris Jensen Aug 13 '21 at 10:45
2

Deleting older data from the current index is not a easy way to do it. You can configure logstash to create new index daily. Then you can access all your data through index patterns or alias.

Then you will be able to delete older indexes without much issue based on the date.

You can automate these using curator - https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/index.html

See this post on configuring logstash to create indexes daily Create a new index per day for Elasticsearch in Logstash configuration

Tanimak
  • 134
  • 1
  • 9
  • Thank you, so i will need to access indexes with logs using some pattern if i will have more than one index? – SolarCore Jun 23 '20 at 21:25
  • 1
    If you are accessing data through Kibana you can easily create an index pattern and use it, You can make all required indexes look like a single entity by using aliases see: https://stackoverflow.com/questions/48907041/what-are-aliases-in-elasticsearch-for – Tanimak Jun 23 '20 at 21:33
  • 1
    Depending on the version of elasticsearch you can use the [Index Lifecycle Management](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) for most of the things that curator was needed in the past. Delete by size or date is one of the things that you can use the Index Lifecycle Management instead of curator. – leandrojmp Jun 24 '20 at 00:43
  • Tanimak and keandrojmp thank you, will look into that solutions too. – SolarCore Jun 24 '20 at 15:29