Removing old indices in elasticsearch

Question

I have the many of my logs indexed in logstash-Year-Week format. That is if i want to delete indices older than a few weeks, how can I achieve that in elasticsearch. Is there an easy, seamless way to do that?

Vineeth Mohan · Accepted Answer · 2015-11-02T12:12:41.340

34

Curator would be an ideal match here. You can find the link here - https://github.com/elastic/curator

A command like below should work just fine -

curator --host <IP> delete indices --older-than 30 --prefix "twitter-" --time-unit days  --timestring '%Y-%m-%d'

You can keep in this in the CRON for removing the indices occasionally.

You can find some examples and docs here - https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html

edited Nov 02 '15 at 12:12

answered Nov 01 '15 at 16:19

Vineeth Mohan

18,633
8
63
77

This was what I was looking for. Do you have documentation on the aplications of curator? – steven johns Nov 02 '15 at 09:52
9

This does NOT work with curator v4 or newer. It requires a config file and an action file, where the curator's action is described. – Eduardo Bezerra Aug 31 '16 at 14:25
See answer by @sachchit-bansal for a working curator 4.2 example – chrisan Mar 24 '17 at 14:08

score 28 · Answer 2 · answered Feb 16 '17 at 08:20

If you are using elasticsearch version 5.x then you need to install the curator version 4.x. You can see the version compatibility and installation steps from the documentation

Once installed. Then just run the command

curator --config path/config_file.yml [--dry-run] path/action_file.yml

Curator provides a dry-run flag to just output what Curator would have executed. Output will be in your log file which you have defined in config.yml file. If not logging key defined in config_file.yml then currator will output to console. To delete the indices run the above command without --dry-run flag

The configuration file config_file.yml is

---
client:
  hosts:
   - 127.0.0.1
  port: 9200
logging:
  loglevel: INFO
  logfile: "/root/curator/logs/actions.log"
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

The action file action_file.yml is

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 7 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:

If you want to delete the indices weekly, monthly, etc automatically. Then just write the bash script like

#!/bin/bash
# Script to delete the log event indices of the elasticsearch weekly

#This will delete the indices of the last 7 days
curator --config /path/config_file.yml /path/action_file.yml

Put a shell script in one of these folders: /etc/cron.daily, /etc/cron.hourly, /etc/cron.monthly or /etc/cron.weekly and your job is done.

NOTE: Make sure to use the correct indentation in your configuration and action files. Otherwise it will not work.

Thanks, this is the current (2017) working version of this answer for curator 4.2 :) — chrisan, Mar 24 '17 at 14:06
That´s the way curator is working! The answer of [Vineeth Mohan](https://stackoverflow.com/users/976646/vineeth-mohan) is outdated from curator 4.x on - and that should apply for most elasticsearch installations right now (where 5.x is current). — jonashackt, May 24 '17 at 13:06

score 18 · Answer 3 · edited Oct 06 '16 at 19:09

I use a bash script, just change the 30 with the # of days you want to keep

#!/bin/bash

# Zero padded days using %d instead of %e
DAYSAGO=`date --date="30 days ago" +%Y%m%d`
ALLLINES=`/usr/bin/curl -s -XGET http://127.0.0.1:9200/_cat/indices?v | egrep logstash`

echo
echo "THIS IS WHAT SHOULD BE DELETED FOR ELK:"
echo

echo "$ALLLINES" | while read LINE
do
  FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' ` 
  if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
  then
    TODELETE=`echo $LINE | awk '{ print $3 }'`
    echo "http://127.0.0.1:9200/$TODELETE"
  fi
done

echo
echo -n "if this make sence, Y to continue N to exit [Y/N]:"
read INPUT
if [ "$INPUT" == "Y" ] || [ "$INPUT" == "y" ] || [ "$INPUT" == "yes" ] || [ "$INPUT" == "YES" ]
then
  echo "$ALLLINES" | while read LINE
  do
    FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' `
    if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
    then
      TODELETE=`echo $LINE | awk '{ print $3 }'`
      /usr/bin/curl -XDELETE http://127.0.0.1:9200/$TODELETE
      sleep 1
      fi
  done
else 
  echo SCRIPT CLOSED BY USER, BYE ...
  echo
  exit
fi

It actually works perfectly and doesn't require installing extra tools like Curator. — Piotr Dawidiuk, Feb 13 '19 at 07:22

score 16 · Answer 4 · answered Nov 16 '20 at 19:53

As of elasticsearch 6.6, Index Lifecycle Management comes included with basic (free) versions elasticsearch, and accomplishes what Curator used to, but in a more graceful way.

The steps below are reproduced without permission from Martin Ehrnhöfer's excellent and concise blog post.

Assumptions (heads up to the copy-pasters):

Your elasticsearch server is accessible at http://elasticsearch:9200
You want your indices to be purged after thirty days (30d)
Your policy name will be created as cleanup_policy
Your filebeat index names begin with filebeat-
Your logstash index names begin with logstash-

1. Create a policy that deletes indices after one month

curl -X PUT "http://elasticsearch:9200/_ilm/policy/cleanup_policy?pretty" \
     -H 'Content-Type: application/json' \
     -d '{
      "policy": {                       
        "phases": {
          "hot": {                      
            "actions": {}
          },
          "delete": {
            "min_age": "30d",           
            "actions": { "delete": {} }
          }
        }
      }
    }'

2. Apply this policy to all existing filebeat and logstash indices

curl -X PUT "http://elasticsearch:9200/logstash-*/_settings?pretty" \
     -H 'Content-Type: application/json' \
     -d '{ "lifecycle.name": "cleanup_policy" }'
curl -X PUT "http://elasticsearch:9200/filebeat-*/_settings?pretty" \
     -H 'Content-Type: application/json' \
     -d '{ "lifecycle.name": "cleanup_policy" }'

3. Create a template to apply this policy to new filebeat and logstash indices

curl -X PUT "http://elasticsearch:9200/_template/logging_policy_template?pretty" \
     -H 'Content-Type: application/json' \
     -d '{
      "index_patterns": ["filebeat-*", "logstash-*"],                 
      "settings": { "index.lifecycle.name": "cleanup_policy" }
    }'

score 7 · Answer 5 · answered Oct 30 '15 at 06:39

7

Take a look at Curator, a tool developed specially for this kind of use case.

A sample command, for the documentation:

curator --host 10.0.0.2 delete indices --older-than 30 --time-unit days \
   --timestring '%Y.%m.%d'

answered Oct 30 '15 at 06:39

Andrei Stefan

51,654
6
98
89

byczk · Answer 6 · 2019-12-10T06:31:58.423

6

you can use curl

 curl -X DELETE http://localhost:9200/filebeat-$(date +"%Y.%m.%d" -d "last Month")

this must to add this command to xxx.sh, and you can create crontab. crontab -e

00 00 * * * /etc/elasticsearch/xxx.sh

this cron will running everyday at 12pm and it will remove old log.

edited Dec 10 '19 at 06:31

answered Dec 09 '19 at 07:48

byczk

81
1
4

1

this is all you need. – visualex Aug 04 '20 at 08:05
Thanks this helped me! in my case, I used like this: ```curl -k -u username:password -X DELETE https://localhost:9200/logstash-$(date +"%Y.%m" -d "-1 month").* ``` I was doing echo to check the date ```echo logstash-$(date +"%Y.%m" -d "-4 month").*``` – Sabuhi Shukurov Jul 25 '22 at 07:14

score 1 · Answer 7 · answered Nov 29 '17 at 12:20

curator_cli delete_indices --filter_list '{"filtertype":"none"}'

will delete all or filter:

 --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":13},{"filtertype":"pattern","kind":"prefix","value":"logstash"}]'

score 0 · Answer 8 · answered Sep 28 '16 at 11:53

yanb (yet another bash)

#!/bin/bash
searchIndex=logstash-monitor
elastic_url=localhost
elastic_port=9200

date2stamp () {
    date --utc --date "$1" +%s
}

dateDiff (){
    case $1 in
        -s)   sec=1;      shift;;
        -m)   sec=60;     shift;;
        -h)   sec=3600;   shift;;
        -d)   sec=86400;  shift;;
        *)    sec=86400;;
    esac
    dte1=$(date2stamp $1)
    dte2=$(date2stamp $2)
    diffSec=$((dte2-dte1))
    if ((diffSec < 0)); then abs=-1; else abs=1; fi
    echo $((diffSec/sec*abs))
}

for index in $(curl -s "${elastic_url}:${elastic_port}/_cat/indices?v" |     grep -E " ${searchIndex}-20[0-9][0-9]\.[0-1][0-9]\.[0-3][0-9]" | awk '{     print $3 }'); do
  date=$(echo ${index: -10} | sed 's/\./-/g')
  cond=$(date +%Y-%m-%d)
  diff=$(dateDiff -d $date $cond)
  echo -n "${index} (${diff})"
  if [ $diff -gt 1 ]; then
    echo " / DELETE"
    # curl -XDELETE "${elastic_url}:${elastic_port}/${index}?pretty"
  else
    echo ""
  fi
done

score 0 · Answer 9 · answered Jan 18 '21 at 12:20

Curator didn't helped me

Nowadays Curator is giving me an error when running it using the below command:

curator --config config_file.yml action_file.yml

Error:

Error: Elasticsearch version 7.9.1 incompatible with this version of Curator (5.2.0)

Cannot find the compatible version of curator with Elasticsearch 7.9.1 and I cannot just upgrade or downgrade the elasticsearch version. So instead I used the @Alejandro's answer and did it using the script below. I modified the script a little bit

Script Solution

#!/bin/bash

# Zero padded days using %d instead of %e
DAYSAGO=`date --date="30 days ago" +%Y%m%d`
ALLLINES=`/usr/bin/curl -s -XGET http://127.0.0.1:9200/_cat/indices?v`
# Just add -u <username>:<password> in curl statement if your elastic search is behind the credentials. Also, you can give an additional grep statement to filter out specific indexes

echo
echo "THIS IS WHAT SHOULD BE DELETED FOR ELK:"
echo

echo "$ALLLINES" | while read LINE
do
  FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | grep -Eo "[0-9]{4}.[0-9]{2}.[0-9]{2}" | sed 's/\.//g'`
  if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
  then
    TODELETE=`echo $LINE | awk '{ print $3 }'`
    echo "http://127.0.0.1:9200/$TODELETE"
  fi
done

echo
echo -n "Y to continue N to exit [Y/N]:"
read INPUT
if [ "$INPUT" == "Y" ] || [ "$INPUT" == "y" ] || [ "$INPUT" == "yes" ] || [ "$INPUT" == "YES" ]
then
  echo "$ALLLINES" | while read LINE
    do
    FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | grep -Eo "[0-9]{4}.[0-9]{2}.[0-9]{2}" | sed 's/\.//g'`
    if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
    then
      TODELETE=`echo -n $LINE | awk '{ print $3 }'`
      /usr/bin/curl -XDELETE http://127.0.0.1:9200/$TODELETE
      sleep 1
      fi
  done
else
  echo SCRIPT CLOSED BY USER, BYE ...
  echo
  exit
fi

score -1 · Answer 10 · answered Jan 24 '20 at 07:13

-1

In my case removing old indices was mandatory since I had Upgraded to 7.5 version from 5.X,

So I followed simple step to clear the indices.

rm -rf /var/lib/elasticsearch/nodes/0/indices/*

answered Jan 24 '20 at 07:13

Sam

1,106
10
14

2

But doing anything bypassing official API is bad. – ipeacocks May 12 '21 at 11:55