1

I have been trying to execute a bash script which works perfect in MacOS but showing weird behavior in windows git bash. Bash script is trying to read from yaml file and print the string for this example.

Git bash version:

GNU bash, version 4.4.23(1)-release (x86_64-pc-msys)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

Yaml File (values.yaml):

apis:
  api1:
    image:
      repository: xxxx.azurecr.io/api1
  api2:
    image:
      repository: xxxx.azurecr.io/api2
  api3:
    image:
      repository: xxxx.azurecr.io/api3

Bash Script:



function pull_apis () {
    local apps=$(yq ".apis | keys" - < ./values.yaml -o json | jq -n 'inputs[]' --raw-output)
    for i in ${apps[@]}
    do
        echo $i
        echo "yq .$i.image.repository - < ./values.yaml"        
        repository=$(yq ".apis.$i.image.repository" - < ./values.yaml)
        echo "----  $repository"
    done
}

pull_apis

It shows below result

api1
.image.repository - < ./values.yaml
----  null
api2
.image.repository - < ./values.yaml
----  null
api3
yq .api3.image.repository - < ./values.yaml
----  xxxx.azurecr.io/api3

Expected result:

api1
yq .api1.image.repository - < ./values.yaml
----  xxxx.azurecr.io/api1
api2
yq .api2.image.repository - < ./values.yaml
----  xxxx.azurecr.io/api2
api3
yq .api3.image.repository - < ./values.yaml
----  xxxx.azurecr.io/api3

I have tried with static array and it works. However, it doesn't work when it reads keys from file.

Could some expert please shade some light on missing part?

Ajay
  • 444
  • 2
  • 9
  • Check your script with [shellcheck.net](https://www.shellcheck.net) to see if there are any obvious issues or warnings. – Paul T. Jul 20 '22 at 04:11
  • @PaulT. As I said, script is already been tested in MacOs and it is working. Issue is with Windows git bash only. I have also check the EOL in yaml file and made it LF to see if that was causing problem. But it is not. However, I will check on your suggested link. – Ajay Jul 20 '22 at 04:20
  • 1
    Your line `echo "yq .$i.image.repository - < ./values.yaml" ` should print `yq ....`, yet the "yq" is gone. This usually indicates that there are some weird characters (i.e. control characters, non-printable, ...) in `$i`. You could inspect the values with `echo "$i" | od -c`. Gut feeling, `$i` is not clean. – Nic3500 Jul 20 '22 at 04:28
  • 1
    I'd put `set -x` at the beginning of the script to get an execution trace as it runs -- that often clarifies what's going wrong. Also, I second the recommendation of [shellcheck.net](https://www.shellcheck.net) even if the script *seems* to be running fine, since it has several quoting problems that *could* cause trouble even if they aren't right now. – Gordon Davisson Jul 20 '22 at 04:33
  • Unless things have changed, MacOS's default shell is not bash – xpusostomos Jul 20 '22 at 06:47
  • @xpusostomos Recent versions of macOS have zsh as the default *interactive* shell, but /bin/sh is bash (running in sh-compatibility mode), so shebangless scripts run in that. Mostly. – Gordon Davisson Jul 20 '22 at 07:15
  • @GordonDavisson thanks for that clarification. Although sh-compatibility removes a heck of a lot more stuff than just shebang. Does mac therefore come with /bin/bash as well? I presume it must. – xpusostomos Jul 20 '22 at 07:26
  • @xpusostomos Yep, /bin/bash is there, it's just stuck on an old version (v3.2.57) to avoid the GPLv3 license that newer versions are under. (That's the real reason they switched the interactive shell to zsh, since they can keep that one up to date.) – Gordon Davisson Jul 20 '22 at 08:24

1 Answers1

1

Thanks @Gordon to point me out to check execution trace. After putting set -x, it shows me the wrong thing happening.

++ pull_apis
++ local apps
+++ yq '.apis | keys' - -o json
+++ jq -n 'inputs[]' --raw-output
++ apps='api1
api2
api3'
++ for i in ${apps[@]}
++ echo $'api1\r'
api1

++ echo $'api1\r' this value was coming with \r. I am able to fix my script after removing it as below,

function pull_apis () {
    local apps=$(yq ".apis | keys" - < ./values.yaml -o json | jq -n 'inputs[]' --raw-output)
    for i in ${apps[@]}
    do
        local param="${i/$'\r'/}"
        echo "$param"        
        echo "yq .$param.image.repository - < ./values.yaml"        
        repository=$(yq ".apis.$param.image.repository" - < ./values.yaml)
        echo "----  $repository"
    done
}

pull_apis
Ajay
  • 444
  • 2
  • 9
  • That's almost certainly due to the values.yaml file having DOS/Windows-style line endings; see ["Are shell scripts sensitive to encoding and line endings?"](https://stackoverflow.com/questions/39527571/are-shell-scripts-sensitive-to-encoding-and-line-endings) for info about how to fix this. BTW, `apps` isn't an array, it's a string with newlines, so `${apps[@]}` isn't doing anything useful. – Gordon Davisson Jul 20 '22 at 07:13
  • @GordonDavisson Yes. I was just trying to see if that make my script work. ${app[@]} is not useful but it works in this case as well. – Ajay Jul 21 '22 at 04:04