2

I have a requirement to inject a conditional execution in my workflow. For ex: If a particular condition is met, then a particular workflow should be executed. If not, a different workflow should be executed.

From my understanding, there is no direct feature in Azkkaban that supports this. Only Oozie supports it. Wanted to know how Azkkaban users are dealing with this? I am sure everyone must be facing this in some form or the other.

Thanks, Kranthi

Kranthi
  • 109
  • 2
  • 7

3 Answers3

1

Azkaban's roadmap says that this feature is there in their future roadmap only and hence this cannot be configured so easily. Only option left is to handle this programatically.

If anyone has better suggestions, please let me know.

thanks, Kranthi

Kranthi
  • 109
  • 2
  • 7
1

FWIW, we do this in our Azkaban projects by checking a condition in a shell script. The script then uses the Azkaban API to trigger another workflow if the condition is met.

#!/bin/bash

# check some condition
check=`…`;

if [ ! -z "$check" ]; then

  # Authenticate ourselves to Azkaban, capturing the session id
  auth=`curl -k -X POST --data "action=login&username=my_user&password=my_password" https://localhost:8443 | jq '.["session.id"]' | sed -e 's/\"//g'`;

  # Run the workflow using the Auth session id, capturing the response
  run=`curl -k --get --data "session.id=$auth" --data 'ajax=executeFlow' --data 'project=my_project' --data "flow=my_flow" https://localhost:8443/executor  | jq '.message'`;

  # If $run response does not contain "successfully" it failed
  if [[ $run = *successfully* ]]; then 
      echo "'$3': RUN STARTED"; 
  else 
      echo "'$3': RUN ERROR"; 
      exit 1; 
  fi; 

else …

You need to be aware that the triggered workflow runs in it's own execution.

Joe Harris
  • 13,671
  • 4
  • 47
  • 54
  • Have you ever looked into Azkaban's TriggerPlugin? I have been looking for an example but don't see anything anywhere. – Felipe Oliveira Feb 16 '16 at 06:07
  • 1
    Not sure what you mean by that TBH. Got a link for me to look at? – Joe Harris Feb 18 '16 at 19:01
  • Trigger plugins are lightly mentioned on Azkaban's documentation: http://azkaban.github.io/azkaban/docs/latest/#plugins but there's not a lot of examples out there. I went over the Azkaban source code and came up with a trigger plugin that does something very similar to what ended up accomplishing with the bash script. https://gist.github.com/feliperazeek/faae2fed6aef780442a4 – Felipe Oliveira Feb 23 '16 at 05:44
0

You can use a trigger plugin in Azkaban to do that. Here's a example of a simple one I came up with: https://gist.github.com/feliperazeek/faae2fed6aef780442a4.

You can implement your own Condition implementation.

Felipe Oliveira
  • 1,041
  • 9
  • 9
  • 1
    Thank you I will write a blog post about it since the documentation on the subject is not great. – Felipe Oliveira Feb 25 '16 at 05:35
  • @ggalmazor not yet but I can help you make sense out of that gist I posted – Felipe Oliveira Mar 24 '16 at 23:56
  • Thanks @FelipeOliveira, that would be great. Anyway, I think my problem is slightly different. I want to run some part of a flow if another part of the flow ends with error (something like a catch-all job). With your example, I guess I could launch a second flow "watching" the first one and waiting for a possible error or success. – ggalmazor Mar 26 '16 at 09:31
  • Right you can probably write a status file if your job errors out and only kick off the second flow if the error file is written. – Felipe Oliveira May 24 '16 at 22:35