trying to create json without jq on linux and send curl results in error

Question

Below is a shell script that I am trying to use to execute dynamic curl calls. The Curl execution calls an API for Kafka. While I know the format of the call is accurate, the url is accurate, and the payload is accurate, I'm getting errors in constructing the payload to send to the API.

Building pipeline for DB1.TB1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   350  100   349  100     1  43059    123 --:--:-- --:--:-- --:--:-- 49857
curl: (6) Could not resolve host: "name"; Unknown error
curl: (6) Could not resolve host: "database.connection.url":"jdbc:sqlserver; Unknown error
curl: (6) Could not resolve host: "database.hostname"; Unknown error

The error indicates that I am pushing a bad payload. I know if the payload is good, then the call should work, because we tested a static version of the payload without the variables. So I am thinking the string injection is the failure here. Somewhere in the function.

The code reads from teh config file, and for each line item, it goes to create the curl call and executes the call.

Bash validators say my code is good, but it's still failing. I'm hoping for a second set of eyes to tell me where I'm wrong. Thank you.

Script.sh

file='Connector.Config'


create_connector () {
              local SHost=${1:?Must Provide Server host}
              local source=${2:?Must provide source logical name}
              local database=${3:?Must provide database}
              local table=${4:?Must provide table}
              local  payload="$(cat <<END
        {
                "name":"${source}_${database}_${table}_source",
                "database.connection.url":"jdbc:sqlserver://${Shost};databaseName=${database};integratedSecurity=true;authenticationScheme=JavaKerberos",
                "database.hostname":"${Shost}",
                "database.server.name":"${Shost}.${table}",
                "database.dbname":"${database}",
                "table.include.list":"dbo.${table}",
                "database.history.kafka.bootstrap.servers":"localhost:9092",
                "database.history.kafka.topic":"${Sname}_${database}_${table}_history",
                "database.user":"sdcservice",
                "connector.class":"io.debezium.connector.sqlserver.SqlServerConnector",
                "database.integratedSecurity":"true",
                "database.authenticationScheme":"JavaKerberos",
                "database.password":"",
                "time.precision.mode": "connect"
        }
END
)"
              local makeConnector=$(curl -i -X PUT -H "Content-Type:application/json" \
                             http://localhost:8083/connectors/${source}_${database}_${table}_source/config \
                                           -d $payload
                                           )

        echo ${makeConnector}
        echo="$makeConnector"
}


while IFS=, read -r source target database table is_deletable action; do
    makeConnector=$(create_connector $Shost $source $database $table)
    echo $makeConnector

Example Config File

Source,Target,Database,Table,Is_deletable,Action
DBServer1,DBServer2,DB,TBL1,true,Add
DBServer1,DBServer2,DB,TBL2,true,Add
DBServer1,DBServer2,DB,TBL3,true,Add

xtrace is your friend. Run `bash -x yourscript`, compare the commands emitted as output to the ones you expect. — Charles Duffy, Mar 03 '22 at 00:30
In this case, `-d $payload` is wrong; it **must** be `-d "$payload"`. http://shellcheck.net/ will automatically detect this and similar issues -- in general, **every** variable expansion should be quoted unless you have a specific, compelling reason you want your code to have the (word-splitting and glob-expansion) behaviors quoting suppresses. — Charles Duffy, Mar 03 '22 at 00:30
(`echo ${makeConnector}` is _also_ wrong; it should be `echo "$makeConnector"` or `echo "${makeConnector}"` for the reasons described in [I just assigned a variable, but `echo $variable` shows something else!](https://stackoverflow.com/questions/29378566/i-just-assigned-a-variable-but-echo-variable-shows-something-else); even that has some buggy corner cases and would be better as `printf '%s\n' "$makeConnector"`, as described in the [unix.se] question [Why is printf better than echo?](https://unix.stackexchange.com/questions/65803/why-is-printf-better-than-echo)) — Charles Duffy, Mar 03 '22 at 00:32
(I'm curious which "bash validator" you used that doesn't catch unquoted variable expansions; for shellcheck -- which is perhaps the most comprehensive and well known of the bunch -- the specific warning thrown is [SC2086](https://github.com/koalaman/shellcheck/wiki/SC2086)). — Charles Duffy, Mar 03 '22 at 00:37
Hi Charles. Thank you so much. this does help ALOT. I used shellchech.net, but it only gave me certain warnings, and not errors. we couldn't tell from this part. — arcee123, Mar 03 '22 at 16:10
It would only be an error if the syntax was invalid to bash. It's a warning because it's valid bash syntax, but does something that you probably don't want. — Charles Duffy, Mar 03 '22 at 16:20

trying to create json without jq on linux and send curl results in error

0 Answers0