Curl post with JSON data fails

Question

I'm trying to use a heredoc to pass in the ONE param I need to make a POST request via curl, this is the error I run into

./scripts/etcd.sh: line 10: warning: here-document at line 10 delimited by end-of-file (wanted `EOF{peerURLs:[http://etcd-$ordinal:2380]}EOF')
./scripts/etcd.sh: line 9: warning: here-document at line 9 delimited by end-of-file (wanted `EOF{peerURLs:[http://etcd-$ordinal:2380]}EOF')
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    42  100    42    0     0  12639      0 --:--:-- --:--:-- --:--:-- 21000
{"message":"unexpected end of JSON input"}

And the 2 lines in the script that trigger it

request_body=$(cat <<EOF{\"peerURLs\":[\"http://etcd-$ordinal:2380\"]}EOF); 
curl http://etcd-0.etcd:2379/v2/members -XPOST -H \"Content-Type: application/json\" --data \"$request_body\";

I've tried all the answers here Using curl POST with variables defined in bash script functions before asking my own question.

Edit:

From the answers and comments below I've tried

curl http://etcd-0.etcd:2379/v2/members -XPOST -H \"Content-Type: application/json\" --data @<(cat <<EOF\n{\"peerURLs\":[\"http://etcd-$ordinal:2380\"]}\nEOF)

And it works, but with giving a similar error

./scripts/etcd.sh: line 12: warning: here-document at line 10 delimited by end-of-file (wanted `EOF')
./scripts/etcd.sh: line 12: warning: here-document at line 10 delimited by end-of-file (wanted `EOF')
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
{"id":"a473da4d8f77b2b0","name":"","peerURLs":["http://etcd-3.etcd:2380"],"clientURLs":[]}

Here-docs don't work like this. The contents of it must start on a new line, and the end marker has to be on a line on its own. Your script thinks that the *whole thing* (`EOF{\"peerURLs\":[\"http://etcd-$ordinal:2380\"]}EOF);`) is your marker. — Benjamin W., Mar 24 '17 at 02:28
You don't want to escape the quotes, either. `curl -H "Content-Type..." --data "$request_body"`. — chepner, Mar 24 '17 at 04:09
Consider using a tool like `jq` to generate your JSON, especially when you are embedding the contents of a parameter. — chepner, Mar 24 '17 at 12:16

score 4 · Accepted Answer · edited May 23 '17 at 11:46

Try:

request_body="{\"peerURLs\":[\"http://etcd-$ordinal:2380\"]}"

Here-docs (<<EOF\n ... \nEOF):

are inherently multi-line constructs (because yours isn't, you're getting the warning)
send their contents to stdin.

Neither aspect is required in your case, where a single-line, expandable string ("...") is sufficient.

Additionally, the way you use quoting in your curl command is broken - see bottom.

If you did want to use a here-doc with a variable, this is the most efficient idiom:

read -d '' -r request_body <<EOF
{"peerURLs":["http://etcd-$ordinal:2380"]}
EOF

^{Note: The closing delimiter, EOF here, must be on its own line, at the very start of the line (no leading whitespace allowed),
and no other characters may follow, not even whitespace and comments.}

read is used to read the here-doc's contents, via stdin (-d '' ensures that the entire here-doc is read as a whole, and -r is used to ensure that the content is read without interpretation of embedded \ chars.)

As you can see, the advantage here is that there's no need to escape embedded " chars. as \".

chepner points out that often you can use here-docs directly, without the need for an auxiliary variable; in this case, @- as the --data argument tells curl to read from stdin:

curl http://etcd-0.etcd:2379/v2/members -XPOST -H 'Content-Type: application/json' \
  --data @- <<EOF
{"peerURLs":["http://etcd-$ordinal:2380"]}
EOF

The above command also shows how to quote the -H option-argument correctly: using unescaped single-quoting to produce a string literal. By contrast, \"Content-Type: application/json\" would create 2 arguments, "Content-Type:and application/json" - with \" turning into embedded " chars.

Similarly, to use your $request_body variable instead of a here-doc, use "$request_body" rather than \"$request_body\".

@Jonathan: It sounds like you're seeing an unrelated error. If you use the here-doc as posted in this answer with `bash`, it should work. Also note that, as chepner has already pointed out, your `curl` command is broken with respect to quoting - see my update. — mklement0, Mar 24 '17 at 12:20
My quoting use is needed, because I'm writing bash commands to a file with cat, within a kubernetes yaml file annotation where the whole block is surrounded by double quotes already, other than thatI agree with you that I'm still very very unfamiliar with heredocs :) — Jonathan, Mar 24 '17 at 12:31
@Jonathan: I see - you should have mentioned that in your question to avoid confusion. — mklement0, Mar 24 '17 at 12:34

Curl post with JSON data fails

1 Answers1