How to get file name dynamically in decision node in OOZIE?

Question

i want check whether the file is exist or not, In HDFS location using oozie batch.

in my HDFS location , in daily base I will get file like "test_08_01_2016.csv","test_08_02_2016.csv" at every day 11PM.

So i want check whether the file exist are after 11.15 PM ,i can check file exist on not using decision node. by using below workflow .

<workflow-app name="HIVECoWorkflow" xmlns="uri:oozie:workflow:0.5">
<start to="CheckFile"/>
<decision name="CheckFile">
     <switch>
        <case to="nextOozieTask">
          ${fs:exists("/user/cloudera/file/input/test_08_01_2016.csv")}
        </case>
        <default to="MailActionFileMissing" />
     </switch>

<action name="MailActionFileMissing" cred="hive2">
    <hive2 xmlns="uri:oozie:hive2-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <jdbc-url>jdbc:hive2://quickstart.cloudera:10000/default</jdbc-url>
        <script>/user/cloudera/email/select.hql</script>
        <file>/user/cloudera/hive-site.xml</file>
    </hive2>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<action name="nextOozieTask" cred="hive2">
    <hive2 xmlns="uri:oozie:hive2-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <jdbc-url>jdbc:hive2://quickstart.cloudera:10000/default</jdbc-url>
        <script>/user/cloudera/email/select1.hql</script>
        <file>/user/cloudera/hive-site.xml</file>
    </hive2>
    <ok to="End"/>
    <error to="Kill"/>
</action>


 <kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="End"/>

but i want to get file name dynamically like for example " filenamt_todaysdate i.e test_08_01_2016.csv".

please help me on this how can i get filename dynamical.

thanks in advance.

RTFM => https://oozie.apache.org/docs/4.1.0/CoordinatorFunctionalSpec.html#a6.7.3._coord:nominalTime_EL_Function — Samson Scharfrichter, Aug 22 '16 at 16:44

score 4 · Accepted Answer · answered Aug 23 '16 at 11:14

The solution for the above question is, we have to get the date value from coordination job like below code ,inside the coordination job.

<property>
    <name>today</name>
    <value>${coord:formatTime(coord:dateTzOffset(coord:nominalTime(), "America/Los_Angeles"), 'yyyyMMdd')}</value>
  </property>

We can check the file exist or not in given HDFS location with the help fs:exists i.e

${fs:exists(concat(concat(nameNode, path),today))}

And in workflow we have to pass the parameter of the coordination job date value “today” like below code

<workflow-app name="HIVECoWorkflow" xmlns="uri:oozie:workflow:0.5">
<start to="CheckFile"/>
<decision name="CheckFile">
     <switch>
        <case to="nextOozieTask">
          ${fs:exists(concat(concat(nameNode, path),today))}
        </case>
         <case to="nextOozieTask1">
          ${fs:exists(concat(concat(nameNode, path),yesterday))}
        </case>
        <default to="MailActionFileMissing" />
     </switch>  </decision>

<action name="MailActionFileMissing" cred="hive2">
    <hive2 xmlns="uri:oozie:hive2-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <jdbc-url>jdbc:hive2://quickstart.cloudera:10000/default</jdbc-url>
        <script>/user/cloudera/email/select.hql</script>
        <file>/user/cloudera/hive-site.xml</file>
    </hive2>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<action name="nextOozieTask" cred="hive2">
    <hive2 xmlns="uri:oozie:hive2-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <jdbc-url>jdbc:hive2://quickstart.cloudera:10000/default</jdbc-url>
        <script>/user/cloudera/email/select1.hql</script>
        <file>/user/cloudera/hive-site.xml</file>
    </hive2>
    <ok to="End"/>
    <error to="Kill"/>
</action><kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="End"/>

in job.properties we can declare all static values like below.

jobStart=2016-08-23T09:50Z
jobEnd=2016-08-23T10:26Z
tzOffset=-8
initialDataset=2016-08-23T09:50Z
oozie.use.system.libpath=True
security_enabled=False
dryrun=True
jobTracker=localhost:8032
nameNode=hdfs://quickstart.cloudera:8020
test=${nameNode}/user/cloudera/email1                
oozie.coord.application.path=${nameNode}/user/cloudera/email1/add-partition-coord-app.xml
path=/user/cloudera/file/input/ravi_

score 0 · Answer 2 · answered Aug 22 '16 at 16:53

0

May be you can write a shell script which does the hdfs file exists check. Upon success return 0 else 1. Based on this rewrite oozie workflow success and error nodes...

answered Aug 22 '16 at 16:53

Despicable me

548
1
9
24

For the record, the same user asked a very similar question a few days ago and got a valid answer (http://stackoverflow.com/questions/39033693/how-to-check-whether-the-file-is-exist-in-hdfs-location-using-oozie-batch). – Samson Scharfrichter Aug 22 '16 at 20:18

How to get file name dynamically in decision node in OOZIE?

2 Answers2