0

I want to delete all logs which are 30 days older from today example logstash filename is below

logstash-2016-11-30

is there any configuration to which I can set a value by which automatically it will remove old data.

and which is the better option configuration (*.yml) files or firing a query.

Mr. A
  • 77
  • 2
  • 8
  • This is the perfect job for the [curator tool](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_delete_indices.html). Check it out. – Val Nov 30 '16 at 09:28
  • If you are using a crontab to launch curator and if you are using the timestring option when launching curator, don't forget that % are considered line break for crontab and have to be escaped like this \% ([cf](http://stackoverflow.com/questions/16238460/why-percent-signs-do-not-work-in-crontab)) – baudsp Nov 30 '16 at 09:45
  • 1
    Possible duplicate of [log rotation script for logstash to purge logs greater than two weeks old](http://stackoverflow.com/questions/31421143/log-rotation-script-for-logstash-to-purge-logs-greater-than-two-weeks-old) – baudsp Nov 30 '16 at 09:51
  • You can check [this](http://stackoverflow.com/questions/31421143/log-rotation-script-for-logstash-to-purge-logs-greater-than-two-weeks-old/) for answers – baudsp Nov 30 '16 at 09:52
  • is there any example to use ttl with index during creation of index or through config file and where to write the ttl in config file, The reason I am looking for this approach is my logstash is not that much big which needs curator tool – Mr. A Dec 02 '16 at 09:57
  • Mr. A, TTL is deprecated in Elasticsearch 2.x, and removed in Elasticsearch 5.0. You must either manage at the index level, or use the delete_by_query plugin. – untergeek Dec 03 '16 at 08:02

2 Answers2

2

For Curator 4.1.2, you can use the next line on crontab:

00 00 * * * curator --config /etc/logstash/curator/curator.yml /etc/logstash/curator/delete

Where /etc/logstash/curator/curator.yml is like

    # Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
   - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  aws_key:
  aws_secret_key:
  aws_region:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

and /etc/logstash/curator/delete is like:

    # Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
  action: delete_indices
  description: >-
  #Delete older than 30 days
  options:
    ignore_empty_list: True
    timeout_override:
    continue_if_exception: False
    disable_action: False #True
  filters:
  - filtertype: pattern
    kind: prefix
    value: logstash-
    exclude:
  - filtertype: age
    source: creation_date
    direction: older
    unit: days
    unit_count: 30
    exclude:

You can create new actions and use other "age" filters like seconds, minutes...

ELosada
  • 33
  • 6
0

Curator 4.2.3 re-introduced the command-line singleton as existed in Curator 3.x. You can read more here.

curator_cli [ARGS] delete-indices --filter_list '...'
untergeek
  • 863
  • 4
  • 13
  • Yes. It's under an Apache 2.0 license. – untergeek Dec 05 '16 at 19:14
  • $ curator_cli curator_cli: command not found but $ curator Usage: curator [OPTIONS] COMMAND [ARGS]... Curator for Elasticsearch indices. See http://elastic.co/guide/en/elasticsearch/client/curator/current How can I install it ? – biolinh Aug 18 '17 at 04:48
  • You're probably installed an older version of curator. Don't just install what comes in the repository for your OS. Elastic provides its own repositories at https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html – untergeek Aug 19 '17 at 19:50