Bash variable is causing disorder to the expected output

Question

I'm trying to create a JSON data and store it to a file and then feed it to the API call I'm gonna make. However, upon formatting the output of my echo command to make a JSON format a variable is causing disarray in the returned output. Below is my script

  cat "${SETTINGS_LIST}" | while read endpoint
  do
    echo "{\"id\": \"$n\", \"method\": GET, \"url\": \"$endpoint\"}"
    let n=n+1
  done

then this is the content of SETTING_LIST variable

/accounting-rules
/aging-buckets
/application-rules
/audit-trail-settings
/batch-aliases
/billing-cycle-types
/billing-list-price-bases
/billing-period-starts
/billing-periods
/billing-rules

And the output is returning like this:

"}id":"0","method":"GET","url":"/accounting-rules
"}id":"1","method":"GET","url":"/aging-buckets
"}id":"2","method":"GET","url":"/application-rules
"}id":"3","method":"GET","url":"/audit-trail-settings
"}id":"4","method":"GET","url":"/batch-aliases
"}id":"5","method":"GET","url":"/billing-cycle-types
"}id":"6","method":"GET","url":"/billing-list-price-bases
"}id":"7","method":"GET","url":"/billing-period-starts
"}id":"8","method":"GET","url":"/billing-periods
"}id":"9","method":"GET","url":"/billing-rules

When I removed the endpoint variable and change it to this, for example

echo "{\"id\": \"$n\", \"method\": GET, \"url\": \"/application-rules\"}"

I'm getting the expected output

{"id": "0", "method": GET, "url": "/application-rules"}
{"id": "1", "method": GET, "url": "/application-rules"}
{"id": "2", "method": GET, "url": "/application-rules"}
{"id": "3", "method": GET, "url": "/application-rules"}
{"id": "4", "method": GET, "url": "/application-rules"}
{"id": "5", "method": GET, "url": "/application-rules"}
{"id": "6", "method": GET, "url": "/application-rules"}
{"id": "7", "method": GET, "url": "/application-rules"}
{"id": "8", "method": GET, "url": "/application-rules"}
{"id": "9", "method": GET, "url": "/application-rules"}

Actions I have done:

1. Downgraded to older git bash version
2. Tried in my Ubuntu WSL but still getting the same error.
3. Tried different way of formatting the output to JSON by using JQ and printf command but issue still persist
4. The script works fine in different machine with same version of BASH and GIT BASH

Does anybody know the cause of this issue and how this could be possibly resolved?

Answer 1

Welcome to a little history lesson on how Unix and Windows thought differently about line endings!

On Unix (and Linux, Ubuntu, ...) a line ending is defined as a single \n character.

On Windows it’s a two-character sequence \r\n.

(For completeness, on old Macs it was a single \r, but that’s mostly irrelevant today.)

You’ve got a lovely, Windows-compatible text file, where lines are delimited with \r\n. Bash however, being a Unix tool, doesn’t handle \r specially and splits the lines only at the \n characters.

Therefore, every instance of your $endpoint variable contains a single \r at the end.

Now, \r is also called “Carriage Return”, from old typewriters, and that’s exactly what Bash’s output routine does, when it encounters a \r: jump back to the start of the current line and replace all characters with what follows after. That’s why you see such strange, garbled output.

Fix: Remove the \r before using the variable. E.g., the answer to this question suggests a solution like this:

cat "${SETTINGS_LIST}" | while IFS= read -r endpoint
do
    endpoint="${endpoint%$'\r'}"
    echo "{\"id\": \"$n\", \"method\": GET, \"url\": \"$endpoint\"}"
    let n=n+1
done

Answer 2

Your settings-list obviously has carriage return characters in it. Replace cat "${SETTINGS_LIST}" by dos2unix < "${SETTINGS_LIST}", i.e.

dos2unix <"${SETTINGS_LIST}"` | while read endpoint

In case you don't have dos2unix yet (I don't remember whether it ships with the Mac), install it from the Appstore.