0

I am new oozie user. Currently I am trying to run a sample bigquery command(e.g: bq ls -p) from a shell script in oozie. But its failing every time. below I have provided the workflow and shell script. I am trying it out in Hortonworks Sandbox and the gcloud is authenticated in hortonworks sandbox box.
I want to know is it not possible to run a bigquery command from oozie? AFAIK Hortonworks sandbox uses same virtualbox as its datanode and jobnode. If I can run then anyone can help me to find the answer - if I am going to run from larger hadoop cluster, do I need to authenticate the gcloud in each node?

Thanks in advance. My workflow xml sample:

<workflow..
   <start to="run_shell" />
      <action name="run_shell" retry-max="2" retry-interval="1">
        <shell xmlns="uri:oozie:shell-action:0.2">
           <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                  <name>mapred.job.queue.name</name>
                  <value>default</value>
                </property>
            </configuration>
            <exec>pl2.sh</exec>
            <argument>/user/bandyoa/AP/</argument>
            <file>${nameNode}/user/bandyoa/AP/pl2.sh#pl2.sh</file> 
            <capture-output/>
        </shell>
        <ok to="end"/>
      <error to="failure_mail"/>
    </action>
</workflo..>

and shell script:

#!/bin/bash
bq ls -p
Christoph
  • 47,569
  • 8
  • 87
  • 187
user1929927
  • 31
  • 1
  • 3
  • what's the error message? – Felipe Hoffa Aug 27 '14 at 20:03
  • Hi Felipe.. I checked the error log .. Actually its not able to find bq commmand.. below the stderr error log Log Type: stderr Log Length: 133 ./pl2.sh: line 2: bq: command not found Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] – user1929927 Aug 28 '14 at 00:52
  • I've never used oozie... no idea how to debug it's loading path. – Felipe Hoffa Aug 28 '14 at 01:43
  • Hi, I am able to run the bq command. The unix $PATH was not refering to the bq. But now I am getting 'Heart Beat' only in the Hortonworks running : 'bq ls -p ' in the shell script from oozie. Is it the problem regarding authentication? FYI if I give 'bq ls -p' directly from command line its giving correct result. Now I am trying to get OAuth 2.0 authentication done. But facing issue: cannot import name SignedJwtAssertionCredentials. (http://stackoverflow.com/questions/14063124/importerror-cannot-import-name-signedjwtassertioncredentials). – user1929927 Aug 29 '14 at 00:13

1 Answers1

0

Copying all the project settings and auth settings from /home/hdfs/.config and /home/hdfs/.bigqueryrc to /home/ and setting them to be readable/writable to all users made Oozie happy for me. Now bq ls returns the list of tables in default dataset.

Nathan Tuggy
  • 2,237
  • 27
  • 30
  • 38
oozie user
  • 1