1

I'm facing an issue while cloning a git repo to the spark dataproc cluster in gcp. I'm using a bash script in initialization action for cloning the repo to the master node of the cluster. Please find the script and the error message below.

pip uninstall -y gcsfs
pip install gcsfs==2021.10.1
ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)
if [[ "${ROLE}" == 'Master' ]]; then
  gcloud config set project myproject-1
  gcloud source repos clone timeseries-forecast --project=myproject-1
  cd /timeseries-forecast
  git checkout origin dev-branch
fi


error message:
/etc/google-dataproc/startup-scripts/dataproc-initialization-script-0: line 9: syntax error near unexpected token `fi'
/etc/google-dataproc/startup-scripts/dataproc-initialization-script-0: line 9: `fi'
Ajay Ganti
  • 35
  • 8
  • As a `bash` script it looks pretty ok even though you could add some error checking. Are you sure it's executed by a `bash` interpreter? Also, `cd /timeseries-forecast` looks fishy. Does `gcloud source repos clone ...` really clone into the root directory? – Ted Lyngmo Feb 26 '22 at 04:42
  • The `if ... then ... fi` syntax looks correct to me; try printing the file with `LC_ALL=C cat -v scriptname.sh` and see if it shows any weird/normally-invisible characters. – Gordon Davisson Feb 26 '22 at 06:09
  • @TedLyngmo yes, the script was executed by a bash interpreter. I see that it clones to the root directory. I checked it by removing the if condition. `gcloud source repos clone ...` clones to the root directory. – Ajay Ganti Feb 26 '22 at 07:02
  • @GordonDavisson I tried running the command you gave to check for any weird/invisible characters. Here is the [output](https://www.toptal.com/developers/hastebin/yudihiliji.sql) for that – Ajay Ganti Feb 26 '22 at 07:10
  • 3
    @AjayGanti Those `^M`s at the end of lines (carriage returns) indicate the script file is in DOS/Windows format, which will cause all sorts of problems (including this one). You need to convert it to unix format, and use an editor that'll save in that format. See ["Are shell scripts sensitive to encoding and line endings?"](https://stackoverflow.com/questions/39527571/are-shell-scripts-sensitive-to-encoding-and-line-endings) for lots more info. – Gordon Davisson Feb 26 '22 at 07:35
  • @GordonDavisson Thanks for the solution. I just changed the script to unix format and it worked like a charm. – Ajay Ganti Feb 26 '22 at 08:01

0 Answers0