Logstash sequential thread execution flow

Question

I made two jdbc sections in my config, but I want always execute first actionA and than actionB, it's important for me. How can I do this?

Here is my config:

input {
    jdbc {
        type => "actionA"
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@mycomp:1522/db1"
        ...
        statement => "SELECT ID FROM my_table WHERE delete='Y'"
       }

    jdbc {
        type => "actionB"
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@mycomp:1522/db1"
        ...
        statement => "UPDATE my_table SET delete='T' WHERE delete='Y'"
       }
}
output {
    stdout { codec => rubydebug }
    if [type] == "actionA" {
        elasticsearch {
            action => "delete"
            index => "contacts"
            document_type => "contact"
            document_id => "%{id}"
            hosts => ["http://localhost:9200"]
        }   
    }
    if [type] == "actionB" { }


}

Thanks in advance

Jay

score 0 · Answer 1 · answered Aug 05 '16 at 07:25

0

You can leverage the schedule parameter and make actionA always run before actionB, e.g. by making actionA run every even minutes (0, 2, 4, 6, ...) and actionB run every odd minutes (1, 3, 5, ...)

input {
    jdbc {
        type => "actionA"
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@mycomp:1522/db1"
        ...
        schedule => "*/2 * * * *" # run every even minutes
        statement => "SELECT ID FROM my_table WHERE delete='Y'"
       }

    jdbc {
        type => "actionB"
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@mycomp:1522/db1"
        ...
        schedule => "1-59/2 * * * *" # run every odd minutes
        statement => "UPDATE my_table SET delete='T' WHERE delete='Y'"
       }
}

answered Aug 05 '16 at 07:25

Val

207,596
13
358
360

It's good variant and i thought about it. But if for some reason SELECT will take more time? How safe is this solution? Maybe it's possible to wait for execution of actionA? – Frank Aug 06 '16 at 17:04
You can always increase the delta between both actions. How often do you need them to run anyway? – Val Aug 07 '16 at 05:29
I need to run it every hour. But i think it's not enterprise solution because there is no guarantee that actionB will be always done after actionA. Actually, in most situations actionA will run very fast and it's not good idea to wait e.g. 10 minutes for actionB. I think the best solution will be writing script that will wait for ending of actionA and then start actionB and start this script with help of CRONE. – Frank Aug 08 '16 at 07:30

score 0 · Answer 2 · answered Oct 17 '17 at 00:33

There is a trick for it.

Put your actions in separate files, so each file looks like this:

# action_X.conf
input {
    jdbc {
        # ...
    }
}
output {
    # ...
}

Then execute logstash manually for each file:

logstash -f action_a.conf
logstash -f action_b.conf

To repeat periodically, you could put the above commands in a shell script and add that script to the crontab.

Logstash sequential thread execution flow

2 Answers2