2

I installed Hadoop on my MacOS. I both tried following the way posted on Hadoop documentaion and by Brew. But whenever I try to run start-dfs.sh or start-yarn.sh or start-all.sh, it just throws the following error:

Error

If you don't want to click the image of screenshot, errors from the Terminal pasted in text here:

Starting resourcemanager
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: `  done < <(for text in "${input[@]}"; do'
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 265: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 273: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/yarn: line 232: hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/yarn: line 233: hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 293: hadoop_add_client_opts: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 300: hadoop_subcommand_opts: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 303: hadoop_generic_java_subcmd_handler: command not found
Starting nodemanagers
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: `  done < <(for text in "${input[@]}"; do'
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 265: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 273: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/yarn: line 232: hadoop_validate_classname: command not found

I have done every step on the Hadoop tutorial.
And I have also tried the solution found on Stackoverflow like using bash instead of sh to run the script and using sudo.

My MacOS version is: 10.14
Hadoop version is: 3.1.1

I did see a similar question on stackoverflow.
But no working solution was answered in that post and the person who asked that question said that he has solved that. I do not have enough reputation to comment and contact that user, masterX.

Daniele Santi
  • 771
  • 3
  • 24
  • 31
Weijian
  • 21
  • 3
  • See my answer here https://stackoverflow.com/a/52531871/2308683 – OneCricketeer Jan 18 '19 at 05:14
  • I checked your answer. I followed your steps but I still had the same error when executing start-dfs.sh. Would you mind giving me more information? – Weijian Jan 18 '19 at 20:47
  • Since you're getting `syntax error`, that indicates you're not running the commands from a Bash shell. And `command not found` means the `$PATH` for those scripts is not setup correctly, for some reason. As shown in my answer there, though, same version of Hadoop installed in Brew, and it worked for me... What you could do is edit the scripts to enable debugging http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html and look for where it is starting to fail. And don't use `sh start-dfs.sh`, you can `chmod +x start-dfs.sh`, then it's just `./start-dfs.sh` from that directory – OneCricketeer Jan 18 '19 at 22:23
  • did you solve it? I meet the same problem – Bargitta May 14 '19 at 12:37

2 Answers2

0

Check permissions on start files. There might be a permission issue. Have you created an additional user for Hadoop jobs?

Sagar Chawla
  • 59
  • 3
  • What permissions do I need to check? Do I have to add an additional user and how? Would you mind giving me more tips? Sorry I am quite a rookie in hadoop. – Weijian Jan 18 '19 at 20:49
0

There can be many causes for that error:

  1. It might be due to a wrong configuration of Hadoop Environmental variables. Try export HADOOP_HOME=/usr/local/Cellar/hadoop/3.1.1/.
  2. It might for permissions: Try sudo chown YOURUSERNAME:YOURUSERNAME /usr/local/Cellar/hadoop/3.1.1/ -R.
Vicente Bolea
  • 1,409
  • 16
  • 39