Running shell script with Oozie

Question

I am trying to run a sh script through Oozie, but I am facing a problem:

Cannot run program "script.sh" (in directory "/mapred/local/taskTracker/dell/jobcache/job_201312061003_0001/attempt_201312061003_0001_m_000000_0/work"): java.io.IOException: error=2, No such file or directory.

Please help me with necessary steps.

run `ls -la` from oozie to prove that it's there. Did you copy it there manually of using oozie via script.sh? — Oleksii, Dec 06 '13 at 22:57
This is similar to http://stackoverflow.com/questions/22391274/oozie-shell-script-action Good explanation given. — Balkrishan Aggarwal, Aug 24 '16 at 04:53

score 7 · Answer 1 · answered May 03 '16 at 12:12

This error is really ambiguous. Here are some issues that have helped me to solve this issue.

-If you are running oozie workflows on a kerberized cluster, make sure to authenticate by passing your Kerberos Keytab as a argument:

...
<shell>
  <exec>scriptPath.sh</exec>
  <file>scriptPath.sh</file>
  <file>yourKeytabFilePath</file>
</shell>
...

-In your shell File (scriptPath.sh), make sure ro remove first line shell reference.

#!usr/bin/bash

indeed, if this shell reference isn't deployed on all data nodes, this can lead to this error code.

Removing #!usr/bin/bash saved my life – Minutis Aug 24 '16 at 12:24 — Minutis, Aug 24 '16 at 12:24

Ignacio Alorre · Answer 2 · 2018-04-16T12:37:18.897

4

I had this same issue because of something really silly. I added a shell block in the workflow, then I selected the corresponding sendMail.sh, but I forgot to add the file sendMail.sh in FILE +.

edited Apr 16 '18 at 12:37

answered Feb 15 '18 at 06:47

Ignacio Alorre

7,307
8
57
94

score 1 · Answer 3 · edited Dec 04 '14 at 19:35

1

workflow.xml :

...
<shell>
  <exec>script.sh</exec>

  <file>scripts/script.sh</file>
</shell>
...

Make sure you have scripts/script.sh in the same folder in hdfs.

edited Dec 04 '14 at 19:35

Community

1
1

answered Dec 14 '13 at 23:32

Oleksii

1,101
7
12

score 1 · Answer 4 · answered Dec 15 '13 at 19:49

An Oozie shell action is executed on a random Hadoop node, i.e. not locally on the machine where the Oozie server is running. As Oleksii says, you have to make sure that your script is on the node that executes the job.

See the following complete examples of executing a shell action and an ssh action:

https://github.com/airawat/OozieSamples/tree/master/oozieProject/workflowShellAction https://github.com/airawat/OozieSamples/tree/master/oozieProject/workflowSshAction

score 1 · Answer 5 · edited Mar 11 '16 at 16:14

1

Try to give full path for HDFS like

<exec>/user/nathalok/run.sh</exec>  
<file>/user/nathalok/run.sh#run.sh</file>

and ensure that in job.properties the path is mentioned correctly for the library and workflow.xml

oozie.libpath=hdfs://server/user/oozie/share/lib/lib_20150312161328/oozie
oozie.wf.application.path=hdfs://bcarddev/user/budaledi/Teradata_Flow

edited Mar 11 '16 at 16:14

YoungHobbit

13,254
9
50
73

answered May 07 '15 at 04:37

user3754136

509
11
25

1

so your line should look like below: /hdfs_path/script.sh /hdfs_path/scripts#script.sh ensure you give # in second line – user3754136 May 07 '15 at 04:38

score 1 · Answer 6 · answered May 11 '15 at 12:11

1

In addition to what others said, this can also be caused by a shell script having wrong line endings (e.g. CRLF for Windows). At least this happened to me :)

answered May 11 '15 at 12:11

LiMuBei

2,868
22
27

score 1 · Answer 7 · answered Jun 30 '17 at 07:53

1

if your shell file exist in your project correlated dir. then it's your shell file format cause this error. you need to convert the format from dos to linux using dos2linux:dos2linux xxxx.sh

answered Jun 30 '17 at 07:53

SHI YU

11
1

Thanks. It helped me. I had to use `set ff=unix` on the script file. – Sivaprasanna Sethuraman May 18 '18 at 10:30

score 0 · Answer 8 · answered Dec 08 '16 at 10:56

Also make sure that the shell scripts are UNIX compliant. If these shell scripts were written in windows environment then it appends windows specific end of lines (EOL) and these scripts are not recognized by the oozie. So you will get "no such file or directory found" in oozie shell actions.

score 0 · Answer 9 · answered Jul 08 '20 at 10:41

workflow.xml would look something like this

<workflow-app name="HiveQuery_execution" xmlns="uri:oozie:workflow:0.5">
<start to="shell-3c43"/>
<kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="shell-3c43">
    <shell xmlns="uri:oozie:shell-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <exec>/user/path/hivequery.sh</exec>
        <file>/user/path/hivequery.sh#hivequery.sh</file>
          <capture-output/>
    </shell>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<end name="End"/>

Job.properties

jobTracker=xxxx.xxx.xxx.com:port
nameNode=hdfs://xxxx.xxx.xxx.com:port

better configure through UI, as suggested above

Running shell script with Oozie

9 Answers9

Linked