3

I would like to use Azkaban for periodic Hive jobs, I have looked through Azkaban documentation, and it seems like by default it doesn't support Hive jobs, do you know how can I use these two together?

I think, I'll have to run Hive jobs as a "command job" available in Azkaban, but maybe someone has worked it out.
I was using Oozie for some time, but It didn't meet my needs.

Thanks.

Srikar Appalaraju
  • 71,928
  • 54
  • 216
  • 264
wlk
  • 5,695
  • 6
  • 54
  • 72

2 Answers2

3

Right now we don't have an easy way. You can certainly hack into the HiveCliDriver and do it from there, but it's suboptimal... Alternatively, just run it as a command line job. We're using a different system at LI. I hope to add this ability pretty quickly, but not sure when I'll have the chance.

Jakob Homan
  • 2,284
  • 1
  • 13
  • 16
  • Any progress on Hive support now? – auny Feb 26 '13 at 14:14
  • Can you give any pointers about the Hack that can be used to run HIVE jobs from Azkaban? – auny Feb 27 '13 at 07:53
  • We did add this and it's being added as part of the Azkaban 2 roll out: https://groups.google.com/d/topic/azkaban-dev/Mc1_gpsQ7uw/discussion. The actual Hive workflow type should hit the github within the next week or so. If you're in Silicon Valley, Richard will be presenting on this work next week: http://www.meetup.com/Open-Data-Bay-Area/events/105644322/?a=co1.3_grp&rv=co1.3 Otherwise, keep an eye on the public list. – Jakob Homan Feb 28 '13 at 07:05
1

As of Azkaban 2.1, Hive jobs are supported. For details, see the documentation.

Michael Hausenblas
  • 13,162
  • 4
  • 52
  • 66