Using curl POST with variables defined in bash script functions

Question

When I echo I get this, which runs when I enter it into the terminal

curl -i \
-H "Accept: application/json" \
-H "Content-Type:application/json" \
-X POST --data '{"account":{"email":"akdgdtk@test.com","screenName":"akdgdtk","type":"NIKE","passwordSettings":{"password":"Starwars1","passwordConfirm":"Starwars1"}},"firstName":"Test","lastName":"User","middleName":"ObiWan","locale":"en_US","registrationSiteId":"520","receiveEmail":"false","dateOfBirth":"1984-12-25","mobileNumber":"9175555555","gender":"male","fuelActivationDate":"2010-10-22","postalCode":"10022","country":"US","city":"Beverton","state":"OR","bio":"This is a test user","jpFirstNameKana":"unsure","jpLastNameKana":"ofthis","height":"80","weight":"175","distanceUnit":"MILES","weightUnit":"POUNDS","heightUnit":"FT/INCHES"}' https://xxx:xxxxx@xxxx-www.xxxxx.com/xxxxx/xxxx/xxxx

But when run in the bash script file, I get this error

curl: (6) Could not resolve host: application; nodename nor servname provided, or not known
curl: (6) Could not resolve host: is; nodename nor servname provided, or not known
curl: (6) Could not resolve host: a; nodename nor servname provided, or not known
curl: (6) Could not resolve host: test; nodename nor servname provided, or not known
curl: (3) [globbing] unmatched close brace/bracket at pos 158

this is the code in the file

curl -i \
-H '"'Accept: application/json'"' \
-H '"'Content-Type:application/json'"' \
-X POST --data "'"'{"account":{"email":"'$email'","screenName":"'$screenName'","type":"'$theType'","passwordSettings":{"password":"'$password'","passwordConfirm":"'$password'"}},"firstName":"'$firstName'","lastName":"'$lastName'","middleName":"'$middleName'","locale":"'$locale'","registrationSiteId":"'$registrationSiteId'","receiveEmail":"'$receiveEmail'","dateOfBirth":"'$dob'","mobileNumber":"'$mobileNumber'","gender":"'$gender'","fuelActivationDate":"'$fuelActivationDate'","postalCode":"'$postalCode'","country":"'$country'","city":"'$city'","state":"'$state'","bio":"'$bio'","jpFirstNameKana":"'$jpFirstNameKana'","jpLastNameKana":"'$jpLastNameKana'","height":"'$height'","weight":"'$weight'","distanceUnit":"MILES","weightUnit":"POUNDS","heightUnit":"FT/INCHES"}'"'" "https://xxx:xxxxx@xxxx-www.xxxxx.com/xxxxx/xxxx/xxxx"

I assume there's an issue with my quotation marks, but I've played with them a lot and I've gotten similar errors. All the variables are defined with different functions in the actual script

Sir Athos · Accepted Answer · 2019-03-05T18:41:25.650

381

You don't need to pass the quotes enclosing the custom headers to curl. Also, your variables in the middle of the data argument should be quoted.

First, write a function that generates the post data of your script. This saves you from all sort of headaches concerning shell quoting and makes it easier to read an maintain the script than feeding the post data on curl's invocation line as in your attempt:

generate_post_data()
{
  cat <<EOF
{
  "account": {
    "email": "$email",
    "screenName": "$screenName",
    "type": "$theType",
    "passwordSettings": {
      "password": "$password",
      "passwordConfirm": "$password"
    }
  },
  "firstName": "$firstName",
  "lastName": "$lastName",
  "middleName": "$middleName",
  "locale": "$locale",
  "registrationSiteId": "$registrationSiteId",
  "receiveEmail": "$receiveEmail",
  "dateOfBirth": "$dob",
  "mobileNumber": "$mobileNumber",
  "gender": "$gender",
  "fuelActivationDate": "$fuelActivationDate",
  "postalCode": "$postalCode",
  "country": "$country",
  "city": "$city",
  "state": "$state",
  "bio": "$bio",
  "jpFirstNameKana": "$jpFirstNameKana",
  "jpLastNameKana": "$jpLastNameKana",
  "height": "$height",
  "weight": "$weight",
  "distanceUnit": "MILES",
  "weightUnit": "POUNDS",
  "heightUnit": "FT/INCHES"
}
EOF
}

It is then easy to use that function in the invocation of curl:

curl -i \
-H "Accept: application/json" \
-H "Content-Type:application/json" \
-X POST --data "$(generate_post_data)" "https://xxx:xxxxx@xxxx-www.xxxxx.com/xxxxx/xxxx/xxxx"

This said, here are a few clarifications about shell quoting rules:

The double quotes in the -H arguments (as in -H "foo bar") tell bash to keep what's inside as a single argument (even if it contains spaces).

The single quotes in the --data argument (as in --data 'foo bar') do the same, except they pass all text verbatim (including double quote characters and the dollar sign).

To insert a variable in the middle of a single quoted text, you have to end the single quote, then concatenate with the double quoted variable, and re-open the single quote to continue the text: 'foo bar'"$variable"'more foo'.

edited Mar 05 '19 at 18:41

answered Jun 10 '13 at 20:54

Sir Athos

9,403
2
22
23

20

"'"$"'" solved my problem where I needed quotes to be not omitted. Thanks. – Uthman May 05 '15 at 00:52
4

This solution works but I think you can emit the extra double quotes surrounding the variable. So instead of this: --data '{"account": {"email": "'"$email"'"} }' you can do this: --data '{"account": {"email": "'$email'"} }' – twistedstream Sep 09 '16 at 15:40
@twistedstream for properly formatted e-mail addresses, yes. If your variable contains anything naughty though (e.g. spaces), omitting the extra quotes will break your script. – Sir Athos Feb 14 '17 at 17:04
how to do the same on Windows? – dKab Mar 13 '17 at 16:19
@dKab if you are on Windows 10, use bash and curl from the Linux Subsystem; otherwise, find one of the Windows ports for curl. – Sir Athos Mar 23 '17 at 21:39
First off, thank you @SirAthos. This saved my butt. Secondly for everyone else that finds this, _make sure you include the __Content-Type:application/json___ It's easy to forget to put in, but it's necessary. – Scrambo May 22 '18 at 17:23
3

Did not work when there was a space after the second EOF: `EOF `. After removing it everything is fine. – Klaas Jul 09 '18 at 16:54
I needed to urlencode, so this worked for me with an endpoint expecting a payload: `--data-urlencode payload="$(generate_post_data)"` – jamis0n Aug 24 '18 at 05:20
I apologize for my ignorance, but where am I defining that `generate_post_data()` function? – dbreaux Mar 04 '19 at 23:40
2

@dbreaux That depends where you run the curl command. If the command is in a script, you simply define the function anywhere above it in that same script. If you are running curl directly from the command line, you have several options, one of which is to type up the function in a new file and then at the command line run `source my_new_file` to define the function in your current environment. After that you can run the curl command as indicated. – Sir Athos Mar 05 '19 at 18:37
Please explain the cat EOF stuff – slashdottir Oct 15 '19 at 22:30
2

@slashdottir That's a bash feature called Here Documents. You can read about it in more detail at [this link](http://tldp.org/LDP/abs/html/here-docs.html) - in particular, check out Example 19-5. There is also already a [full question about it](https://stackoverflow.com/questions/2953081/how-can-i-write-a-heredoc-to-a-file-in-bash-script) here on SO. – Sir Athos Oct 17 '19 at 18:53

score 197 · Answer 2 · edited Aug 29 '23 at 21:03

197

Solution tested with https://httpbin.org/ and inline bash script
1. For variables without spaces in it i.e. 1:
Simply add ' before and after $variable when replacing desired string

for i in {1..3}; do \
  curl -X POST -H "Content-Type: application/json" -d \
    '{"number":"'$i'"}' "https://httpbin.org/post"; \
done

2. For input with spaces:
Wrap variable with additional " i.e. "el a":

declare -a arr=("el a" "el b" "el c"); for i in "${arr[@]}"; do \
  curl -X POST -H "Content-Type: application/json" -d \
    '{"elem":"'"$i"'"}' "https://httpbin.org/post"; \
done

edited Aug 29 '23 at 21:03

desertnaut

57,590
26
140
166

answered Mar 30 '17 at 10:52

pbaranski

22,778
19
100
117

2

Does not work for when when `$i` contains spaces. :( – Vasyl Boroviak Apr 03 '17 at 09:15
Can you post an example? – pbaranski Apr 03 '17 at 09:18
1

Sure. `i="a b"` instead of for-loop – Vasyl Boroviak Apr 03 '17 at 09:25
7

I found that the accepted and the second voted answer does not work in `/bin/sh`. However, this answer did the trick. And it's much simpler than the other answers. Thank you so much! I've edited your answer with some nicer line wrapping formatting. Otherwise, it's hard to spot the brilliance. Cheers mate – Vasyl Boroviak Apr 04 '17 at 00:36
Am I correct that the reason the quoting works is because the single-quotes are paired like so: `'{"number":"'` # the first pair of single-quotes, this has the opening double-quote for json syntax. `$i` # bash does variable substitution. `'"}'` # the next pair of single-quotes, this has the closing double-quote for json syntax. – user3053230 Apr 27 '17 at 06:29
1

Thanks a lot @pbaranski you saved lot of my time – sudhir tataraju Oct 17 '19 at 14:08
3. for input with multi var which has spaces:`'{"text": "1.'"$var1"'\n2.'"$var2"'"}'` – AATHITH RAJENDRAN Oct 19 '20 at 08:21
simply worked for me. Thanks a million – Oussama Boumaad Jan 07 '22 at 19:58

score 56 · Answer 3 · edited Aug 29 '23 at 21:03

56

Curl can post binary data from a file so I have been using process substitution and taking advantage of file descriptors whenever I need to post something nasty with curl and still want access to the vars in the current shell. Something like:

curl "http://localhost:8080" \
-H "Accept: application/json" \
-H "Content-Type:application/json" \
--data @<(cat <<EOF
{
  "me": "$USER",
  "something": $(date +%s)
  }
EOF
)

This winds up looking like --data @/dev/fd/<some number> which just gets processed like a normal file. Anyway if you wanna see it work locally just run nc -l 8080 first and in a different shell fire off the above command. You will see something like:

POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.43.0
Accept: application/json
Content-Type:application/json
Content-Length: 43

{  "me": "username",  "something": 1465057519  }

As you can see you can call subshells and whatnot as well as reference vars in the heredoc.

edited Aug 29 '23 at 21:03

desertnaut

57,590
26
140
166

answered Jun 04 '16 at 16:37

Che Ruisi-Besares

711
6
4

2

The other answer didn't work for me as I was trying to invoke it in an alert from Zabbix. This one solves it perfectly and is more clean. – 0rkan Nov 22 '16 at 14:54
But what if you put the code in a bash function : myFunction () { .... } ? – Hanynowsky Dec 08 '16 at 08:43
1

it's worth to note that this recipe works only if the script is copied verbatim (i.e. no reformatting EOF, braces etc.) – Vader B Nov 26 '17 at 06:17
The most elegant answer I would say in terms of simplicity, readability.. – harshavmb May 12 '21 at 13:11

Shakiba Moshiri · Answer 4 · 2022-07-30T07:47:20.777

We can assign a variable for curl using single quote ' and wrap some other variables in

a single quote => ' $variable '
a double-quote single-quote => "' $variable '"
a single+double+single quote => '"' $variable '"'

Lets test each case, but first watch out for this catch that if we use a single quote ' for variable assignment, that variable is not evaluated.

watch out

Please notice the assignment is done by a single quote CURL_DATA='content'

cmd='ls'

CURL_DATA='{
    "cmd": "$cmd",    <===== our variable
    "args": [ "-la" , "/tmp" ],
    "options": {
        "cwd": "/tmp"
    },
   "type": "sync"
}';

echo "$CURL_DATA";

Will give us

{
    "cmd": "$cmd",    <===== we need ls not $cmd
    "args": [ "-la" , "/tmp" ],
    "options": {
        "cwd": "/tmp"
    },
   "type": "sync"
}

a single quote `'` $variable `'`

The value of a variable is evaluated
neither single quote ' nor applies double one "

cmd='ls'

CURL_DATA='{
    "cmd": '$cmd',    <===== our variable
    "args": [ "-la" , "/tmp" ],
    "options": {
        "cwd": "/tmp"
    },
   "type": "sync"
}';

echo "$CURL_DATA";

Will give us:

{
    "cmd": ls,    <===== neither 'ls' nor "ls", just ls
    "args": [ "-la" , "/tmp" ],
    "options": {
        "cwd": "/tmp"
    },
   "type": "sync"
}

a double-quote single-quote `"'` $variable `'"`

variable is evaluated
will be surrounded by a double quote "

cmd='ls'

CURL_DATA='{
    "cmd": "'$cmd'",    <===== our variable
    "args": [ "-la" , "/tmp" ],
    "options": {
        "cwd": "/tmp"
    },
   "type": "sync"
}';

echo "$CURL_DATA";

Will give us

{
    "cmd": "ls",    <===== we have double quote " variable "
    "args": [ "-la" , "/tmp" ],
    "options": {
        "cwd": "/tmp"
    },
   "type": "sync"
}

a single+double+single quote => `'"'` $variable `'"'`

variable is evaluated
will be surrounded by a single quote '

cmd='ls'

CURL_DATA='{
    "cmd": '"'$cmd'"',
    "args": [ "-la" , "/tmp" ],
    "options": {
        "cwd": "/tmp"
    },
   "type": "sync"
}';

echo "$CURL_DATA";

Will give us

{
    "cmd": 'ls',    <===== we have a single quote ' variable '
    "args": [ "-la" , "/tmp" ],
    "options": {
        "cwd": "/tmp"
    },
   "type": "sync"
}

summary

# no quote at all
$cmd => $cmd

# a double quote (" $variable ")
"$cmd" => "$cmd"

# a single quote (' $variable ')
'$cmd' => ls

# a single quote + a double quote ("' $variable '")
"'$cmd'" => "ls"

# a single-double-single quote ('"' $variable '"')
'"'$cmd'"' => 'ls'

which one we should use?

Since JSON needs a double quote " for its key or value we can use :

a double-quote single-quote "' $variable '"

curl

cmd='ls'

CURL_DATA='{
    "cmd": "'$cmd'",
    "args": [ "-la" , "/tmp" ],
    "options": {
        "cwd": "/tmp"
    },
   "type": "sync"
}';

echo "$CURL_DATA" | jq '.'

curl --data "$CURL_DATA" -X POST localhost:3232/cmd | jq '.'

NOTE:
The equivalent of ' for a variable evaluation is '" which means instead of using '$cmd' we can use '"$cmd"' and it gives us ls neither with a single quote nor a double quote , but it gets more confusing if we needed to apply for curl since we need a double quoted result "ls" and would have to wrap it in another double quote => "'"

This code works well, but the above is more readable

cmd='ls'

CURL_DATA='{
    "cmd": "'"$cmd"'",     <===== our variable
    "args": [ "-la" , "/tmp" ],
    "options": {
        "cwd": "/tmp"
    },
   "type": "sync"
}';

echo "$CURL_DATA" | jq '.'

curl --data "$CURL_DATA" -X POST localhost:3232/cmd | jq '.'

Will give us:

{
    "cmd": "ls",    <===== result
    "args": [ "-la" , "/tmp" ],
    "options": {
        "cwd": "/tmp"
    },
   "type": "sync"
}

finally

We can use either of:

    "cmd": "'$cmd'",    <===== will be: "ls"

or

    "cmd": "'"$cmd"'",    <===== will be: "ls"

and "$CURL_DATA" as a normal variable

curl --data "$CURL_DATA" -X POST localhost:3232/cmd

For some reason, shell script was escaping quotes, the right combination of quotes for the placeholders did it for me. `some_body='{"query":"/v1/'$some_id'","body":""}' curl --data "$some_body" 'https://some.url'` — kisna, Jul 20 '21 at 05:31

score 13 · Answer 5 · answered Jan 17 '18 at 03:42

13

A few years late but this might help someone if you are using eval or backtick substitution:

postDataJson="{\"guid\":\"$guid\",\"auth_token\":\"$token\"}"

Using sed to strip quotes from beginning and end of response

$(curl --silent -H "Content-Type: application/json" https://${target_host}/runs/get-work -d ${postDataJson} | sed -e 's/^"//' -e 's/"$//')

answered Jan 17 '18 at 03:42

111

1,788
1
23
38

simple solution, helped a lot, thanks! – user2809176 Dec 16 '22 at 20:28

score 11 · Answer 6 · answered Sep 09 '19 at 22:47

Here's what actually worked for me, after guidance from answers here:

export BASH_VARIABLE="[1,2,3]"
curl http://localhost:8080/path -d "$(cat <<EOF
{
  "name": $BASH_VARIABLE,
  "something": [
    "value1",
    "value2",
    "value3"
  ]
}
EOF
)" -H 'Content-Type: application/json'

score 5 · Answer 7 · edited Aug 29 '23 at 21:02

5

the info from Sir Athos worked perfectly!!

Here's how I had to use it in my curl script for couchDB. It really helped out a lot. Thanks!

bin/curl -X PUT "db_domain_name_:5984/_config/vhosts/$1.couchdb" -d '"/'"$1"'/"' --user "admin:*****"

edited Aug 29 '23 at 21:02

desertnaut

57,590
26
140
166

answered Apr 07 '14 at 19:54

score 5 · Answer 8 · answered Sep 05 '19 at 06:43

Existing answers point out that curl can post data from a file, and employ heredocs to avoid excessive quote escaping and clearly break the JSON out onto new lines. However there is no need to define a function or capture output from cat, because curl can post data from standard input. I find this form very readable:

curl -X POST -H 'Content-Type:application/json' --data '$@-' ${API_URL} << EOF
{
  "account": {
    "email": "$email",
    "screenName": "$screenName",
    "type": "$theType",
    "passwordSettings": {
      "password": "$password",
      "passwordConfirm": "$password"
    }
  },
  "firstName": "$firstName",
  "lastName": "$lastName",
  "middleName": "$middleName",
  "locale": "$locale",
  "registrationSiteId": "$registrationSiteId",
  "receiveEmail": "$receiveEmail",
  "dateOfBirth": "$dob",
  "mobileNumber": "$mobileNumber",
  "gender": "$gender",
  "fuelActivationDate": "$fuelActivationDate",
  "postalCode": "$postalCode",
  "country": "$country",
  "city": "$city",
  "state": "$state",
  "bio": "$bio",
  "jpFirstNameKana": "$jpFirstNameKana",
  "jpLastNameKana": "$jpLastNameKana",
  "height": "$height",
  "weight": "$weight",
  "distanceUnit": "MILES",
  "weightUnit": "POUNDS",
  "heightUnit": "FT/INCHES"
}
EOF

score 1 · Answer 9 · answered Dec 20 '22 at 20:00

This script will make a POST request to the specified URL with the specified JSON body, and it will output the response from the server to the console:

#!/bin/bash

# Set the URL to send the request to
url='http://example.com/endpoint'

# Set the JSON body of the request
json_data='{"key1": "value1", "key2": "value2"}'

# Make the POST request with the JSON body
response=$(curl -X POST -H "Content-Type: application/json" -d "$json_data" "$url")

echo "Response from server: $response"

how to use variable in json_data tho? – Tilo Aug 29 '23 at 19:57 — Tilo, Aug 29 '23 at 19:57

score 0 · Answer 10 · answered Jan 14 '21 at 16:30

0

Putting data into a txt file worked for me

bash --version
GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu)
 cat curl_data.txt 
 {  "type":"index-pattern", "excludeExportDetails": true  }

curl -X POST http://localhost:30560/api/saved_objects/_export -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d "$(cat curl_data.txt)" -o out.json

answered Jan 14 '21 at 16:30

bhargav joshi

329
3
6

1

Of course that works, but then you have a pesky temporary file to clean up afterwards. This is only really useful on Windows, where you can't reliably interpolate variables into strings. – tripleee Apr 08 '21 at 16:20

score 0 · Answer 11 · answered Jun 18 '22 at 08:08

0

jo might help:

$ export A=this B=that C=foo D=bar
$ jo -p a=$A b=$B nested=$(jo c=$C d=$D)
{
   "a": "this",
   "b": "that",
   "nested": {
      "c": "foo",
      "d": "bar"
   }
}

answered Jun 18 '22 at 08:08

pointyhat

568
5
16

Using curl POST with variables defined in bash script functions

11 Answers11

watch out

a single quote `'` $variable `'`

a double-quote single-quote `"'` $variable `'"`

a single+double+single quote => `'"'` $variable `'"'`

summary

which one we should use?

curl

finally

Linked

Related

Using curl POST with variables defined in bash script functions

11 Answers11

watch out

a single quote ' $variable '

a double-quote single-quote "' $variable '"

a single+double+single quote => '"' $variable '"'

summary

which one we should use?

curl

finally

Linked

Related

a single quote `'` $variable `'`

a double-quote single-quote `"'` $variable `'"`

a single+double+single quote => `'"'` $variable `'"'`