How to get HTTP status errors from cURL 8.0?

Question

When testing some REST API, I'm using a cURL command. However when the request failed, it seems that cURL does not indicate that there was a non-success status code (HTTP/2 405 in my case); instead I just see the response (which is a JSON "messages" array).

As it would require "quite a lot of AI" to deduce from the contents of the message array that there was an error, I wonder how I can reliably detect the non-success status code when using cURL.

Maybe it could even be a bug in cURL (regarding use of HTTP/2).

Example session (I deliberately mad a request error to demonstrate):

> curl -u monitor  'https://redacted.server.name/api/status/get-token'
Enter host password for user 'monitor':

{
  "messages": [
    "The global command get-token requires to use PUT"
  ]
}
> echo $?
0

The question was not "how to *see*", but "how to *get*", meaning how to test using a program. If I would have to "grep" the headers, wouldn't that make cURL rather useless? Finally how do you think I found out that there's a "HTTP/2 405" status? ;-) — U. Windl, Aug 21 '23 at 07:28
Then I suggest you rephrase your question. CURL returns `string` output and not `boolean` or `int`, you need to parse its returned content for that. CURL is not really made for testing purposes in that way that your phrasing. — Anuga, Aug 21 '23 at 07:37
Dupe https://stackoverflow.com/questions/38906626/curl-to-return-http-status-code-along-with-the-response/ — dave_thompson_085, Aug 21 '23 at 09:39
@Anuga As the HTTP status codes are well-defined, cURL *could* map them to exit status. HTTP does not require to send any more error details along with the status code, so relying on the response data only may be unreliable. For real use I won't use cURL or BASH. — U. Windl, Aug 22 '23 at 06:42
Does this answer your question? [Curl to return http status code along with the response](https://stackoverflow.com/questions/38906626/curl-to-return-http-status-code-along-with-the-response) — hanshenrik, Aug 24 '23 at 14:06

hanshenrik · Accepted Answer · 2023-08-24T04:53:41.743

1

curl --write-out '\nresponse_code=%{response_code}\n' 'https://redacted.server.name/api/status/get-token'

should end with

response_code=405

if you need it in a variable i suppose you could do

RESPONSE=$(curl --write-out '\n%{response_code}' 'https://redacted.server.name/api/status/get-token')
RESPONSE_CODE=$(tail -1 <<<$RESPONSE)
RESPONSE=$(echo "$RESPONSE" | head --lines=-1)

now echo "$RESPONSE_CODE" should give you 405 and echo "$RESPONSE" should give you the original http response body...

Btw seems you're approaching the complexity where using bash no longer makes sense, maybe try switching to a better scripting language like PHP, Python, or Perl? They all have good libcurl bindings.

edited Aug 24 '23 at 04:53

answered Aug 21 '23 at 08:40

hanshenrik

19,904
4
43
89

2

HTTP body doesn't have to end with NL, and for JSON it often doesn't; safer to force \n _before_ RESPONSE_CODE as well. And can just write `\n%{response_code}\n` then do `RESPONSE_CODE=$(tail -1 <<<$RESPONSE)` instead of eval. Or use no newlines and take 3 _chars_: https://stackoverflow.com/questions/54038132/obtain-status-code-and-response-body-from-curl-request – dave_thompson_085 Aug 21 '23 at 09:30
@dave_thompson_085 seems curl hardcodes a newline between the HTTP Body and `--write-out` - great idea to use `RESPONSE_CODE=$(tail -1 <<<$RESPONSE)` tho – hanshenrik Aug 21 '23 at 09:53
I agree that cURL with BASH has limits, but quite a lot of REST API docs refer to using cURL, and just to verify the principle, I used cURL and BASH. Once it's at parsing or creating JSON, it's better to leave BASH, however. – U. Windl Aug 22 '23 at 06:38
It doesn't add newline for me; please try [this test server I set up temporarily on AWS](http://3.135.217.83) (either approach, but especially yours) – dave_thompson_085 Aug 24 '23 at 02:44
@dave_thompson_085 dang you're right, it doesn't add a newline, and the code is vulnerable. thanks for pointing it out, code updated. – hanshenrik Aug 24 '23 at 05:06
Actually I implemented a similar solution, however there are some pitfalls: First when using a proxy that returned a "503", I just got an empty response, so I needed an error code. First surprise was that `%{http_code` showed as `000`, so I had to examine `%{http_connect}` (that was `503`), too. And for completeness I wanted to capture `%{errormsg}` too. – U. Windl Aug 25 '23 at 09:38
@U.Windl neat. if you're feeling charitable, maybe post your solution as an answer here? might help someone else ^^ – hanshenrik Aug 25 '23 at 09:48
@hanshenrik See https://stackoverflow.com/a/76990503/6607497 – U. Windl Aug 28 '23 at 06:37

score 0 · Answer 2 · answered Aug 28 '23 at 06:37

As the response is JSON, I decided to format the extra status codes JSON-like, too (based on the suggestion in https://stackoverflow.com/a/76943672/6607497). I use these extra options to curl:

-w \
'"_conn%{urlnum}": %{http_connect}\n'\
'"_code%{urlnum}": %{http_code}\n'\
'"_emsg%{urlnum}": "%{errormsg}"\n'

With curl's response in RESP I transform the result (all JSON values are assumed to be either numbers or strings on a single line). FILE is some temporary filename:

sed -n -e 's/^ *"\([^"]\+\)": *"\?\([^"]\+\)"\?,\?$/\1=\2/p' > "$FILE"

To lookup values I added this helper:

get_info()
{
    sed -n -e 's/^'"$1"'=\(.\+\)$/\1/p' "$FILE"
}

So finally the code to check the result (with a squid proxy present) looks like this:

STATUS=$(get_info _code0)
[ -z "$STATUS" -o "$STATUS" = 000 ] && STATUS=$(get_info _conn0)
if [ "$STATUS" = 200 ]; then
    # success...
    rm "$FILE"
else
    EMSG=$(get_info _emsg0)
    [ -n "$EMSG" ] && STATUS="$STATUS $EMSG"
    emsg "$0: failed with status \"$STATUS\":"
    emsg "$RESP"  # show the complete response for debugging purposes
    rm "$FILE"
fi

emsg just writes to STDERR:

emsg()
{
    echo "$@" >&2
}

How to get HTTP status errors from cURL 8.0?

2 Answers2