Hadoop fs command working on the command line but not in shell script

Question

I have a bash shell script that contains the following command:

hadoop fs -get ${WORKING_DIRECTORY}${OUTPUT_FILE}

which resolves into

hadoop fs -get /tmp/out.csv

When I run the shell script, this command produces no output, and the file is not transferred to the local instance. However, if I run the command hadoop fs -get /tmp/out.csv directly on the shell, it works.

What am I missing?

Are you sure the env vars are available? Try to `echo` those on the script. — Henry Harutyunyan, Jun 24 '20 at 14:15
@HenryHarutyunyan I'm positive. They're defined only a few lines above and it's confirmed with `echo`s — Cam, Jun 24 '20 at 14:20
@Cam mmhm is the path correct? Maybe an issue with the slash. Also, does the script have proper access rights to that csv file? — Henry Harutyunyan, Jun 24 '20 at 14:34

score 1 · Answer 1 · answered Jun 24 '20 at 14:23

The issue could be with your script not seeing environment variables such as HADOOP_CONF, and using local filesystem instead of HDFS.

Make sure that .bashrc and .bash_profile are setup properly.

Also you should put set -x2 on top of your script just to make sure the variables are resolved the way you think they should.

Prefer running your scripts in strict mode 1 at least when debugging, will save you a lot of time.

Hadoop fs command working on the command line but not in shell script

1 Answers1