1

simple question here with maybe a complex answer? I have several logstash docker containers running on the same host using the JDBC plugin. Each of them does work every minute. For example:

input {
    jdbc {
        jdbc_driver_library => "/usr/share/logstash/bin/mysql-connector-java-8.0.15.jar"
        jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
        # useCursorFetch needed cause jdbc_fetch_size not working??
        # https://discuss.elastic.co/t/logstash-jdbc-plugin/84874/2
        # https://stackoverflow.com/a/10772407
        jdbc_connection_string => "jdbc:mysql://${CP_LS_SQL_HOST}:${CP_LS_SQL_PORT}/${CP_LS_SQL_DB}?useCursorFetch=true&autoReconnect=true&failOverReadOnly=false&maxReconnects=10"
        statement => "select * from view_elastic_popularity_scores_all where updated_at > :sql_last_value"
        jdbc_user => "${CP_LS_SQL_USER}"
        jdbc_password => "${CP_LS_SQL_PASSWORD}"
        jdbc_fetch_size => "${CP_LS_FETCH_SIZE}"
        last_run_metadata_path => "/usr/share/logstash/codepen/last_run_files/last_run_popularity_scores_live"
        jdbc_page_size => '10000'
        use_column_value => true
        tracking_column => 'updated_at'
        tracking_column_type => 'timestamp'
        schedule => "* * * * *"
    }
}

Notice the schedule is * * * * *? That's the crux. I have a box that's generally idle for 50 seconds out of every minute, then it's working its ass off for x seconds to process data for all 10 logstash containers. What'd be amazing is if I could find a way to splay the time so that the 10 containers work on independent schedules, offset by x seconds.

Is this just a dream? Like world peace, or time away from my kids?

Thanks

timsabat
  • 2,208
  • 3
  • 25
  • 34

1 Answers1

1

I believe rufus cronlines (which is what the schedule option is) can specify seconds.

'13 0 22 * * 1-5' means every day of the week at 22:00:13

Badger
  • 3,943
  • 2
  • 6
  • 17
  • 1
    Am I understanding the syntax correctly? Would this give me the 30th second of every minute? `Rufus::Scheduler.parse('30 1 * * * *')` – timsabat Jul 18 '19 at 20:26
  • 1
    I believe that 'schedule => "30 * * * * *"' would give you the 30th second of every minute. – Badger Jul 18 '19 at 22:12