Why does curl not return a value in bash script?

Question

My task is very simple - nevertheless I have been sitting already for hours and have no idea why it doesn't work.

In linux bash script I want to get the result of a webservice call with curl. I'm not interested in the content, only the status code:

#!/bin/bash
set -euo pipefail # put bash in "strict mode"

echo "Before"
response2=$(curl  -o /dev/null -s  -w '%{http_code}'  -u username:password-X POST https://xxxxx.yyyyy.at:8081/MyPath/service/rest/crypto/encrypt -H 'Content-Type: application/json' -d '{"deviceId": "ABCD","payload": "ABCD"}')
echo "after"

It works when there is a valid request

Before...
200

Also, when the path of the service is wrong, it gives http error code

Before...
500

But when the host is wrong (not existent hostname) I get

Before...

and the script terminates (although the call is from a looping menue).

Why is this the case?

The manual call of curl with same parameters gives

as output, so why this output is not displayed in my script?

A reproducable example is (server name not existing):

#!/bin/bash
set -euo pipefail


#- Check kms
f_check_kms(){

    echo "Before..."
    response2=$(curl  -o /dev/null -s  -w '%{http_code}'  -u user:xxxx -X POST https://xxxx.xxx.intra.graz.at:8081/ATM-KeyManagement-REST-Service/service/rest/crypto/encryptUCast -H 'Content-Type: application/json' -d '{"deviceId": "SAG0530000016261", "encryptionSuite": "DLMS_SUITE_0", "securityMode": "AUTHENT_AND_ENCRYPT", "roleId": "001","initialVector": "4D4D4D0000BC614E01234567","payload": "ABCD","usedGuek":"NO","usedGak":"NO"}')
    
    echo "$response2"

}



f_check_kms

Let me guess -- you're running your script with `set -e` (`#!/usr/bin/bash -e` or similar)? [Don't do that](https://mywiki.wooledge.org/BashFAQ/105#Exercises). If you handle nonzero exit statuses _by hand_, you make all the decisions about how to manage them, so you don't get decisions that surprise you getting made behind your back. — Charles Duffy, Feb 07 '23 at 15:14
And it'd be better if we didn't _have_ to guess at your shebang or how you were invoking the script -- if you were providing a proper [mre] we could run _without changes_ to see the same behavior. — Charles Duffy, Feb 07 '23 at 15:16
#!/bin/bash set -euo pipefail is on the top. To be honest, this is my first contact with bash and I have just to make a slight add-on to an existing script. — MichaelW, Feb 07 '23 at 15:16
Yeah. The people who call `set -euo pipefail` a "strict mode" have done a lot of harm; as you can see if you follow the link above (or the reference of behavior in historical shells at https://www.in-ulm.de/~mascheck/various/set-e/), it causes a _lot_ of unintuitive, surprising behaviors, not to mention behaves in incompatible ways across both different shells and different releases of the same shell. Needing to write out error handling by hand isn't fun, but it's the right thing to do if you want your code's behavior to make sense. — Charles Duffy, Feb 07 '23 at 15:17
Does it mean that by using set -e whenever a subshell encounters an error, the whole script terminates? My task is only a tiny part of the existing script, so I better don't want to remove this option. Is there an alternative? Why is there an error at all? Just because curl returns 0 ? — MichaelW, Feb 07 '23 at 15:23
0 is what it would return on _success_. Returning anything other than 0 is an error. — Charles Duffy, Feb 07 '23 at 15:24
Anyhow -- I added an answer describing how to work around the behavior. ("Whenever a subshell encounters an error" isn't always true, but it's _sometimes_ true; show me a given piece of code and specify an exact shell version and I can explain why you get the behavior you do, but it's hard to state a hard-and-fast rule -- indeed, the set of overlapping rules and exceptions to those rules is much of why `set -e` is such an awful mess). — Charles Duffy, Feb 07 '23 at 15:26
Essentially a duplicate of https://stackoverflow.com/questions/19622198/what-does-set-e-mean-in-a-bash-script — tripleee, Feb 07 '23 at 15:50
For the "why" part, I suppose it could simply be "there is no HTTP status code from the HTTP server when you could not connect to the HTTP server". — tripleee, Feb 07 '23 at 15:52
@tripleee, but curl _does_ print `000` in that case; this is why the OP is so confused. — Charles Duffy, Feb 07 '23 at 15:55

score 3 · Accepted Answer · edited Feb 07 '23 at 15:58

You're running your script with set -e to make the shell interpreter exit when any¹ unchecked² command exits with a nonzero status, and when you provide an invalid hostname, curl exits with a nonzero exit status.

Because you're passing -s for silent mode, it doesn't print any error messages about this (you asked it not to!). It does still print the http_code you asked for, but because the script exits, the echo "after" is never reached, and whatever other code you're relying on to print the contents of the response2 variable is likewise never reached.

Suppressing this is as simple as adding a conditional to the end, like the || : sequence below:

response2=$(curl -o /dev/null -s  -w '%{http_code}' -u username:password \
  -X POST https://xxxxx.yyyyy.at:8081/MyPath/service/rest/crypto/encrypt \
  -H 'Content-Type: application/json' \
  -d '{"deviceId": "ABCD","payload": "ABCD"}' \
) || : "Ignoring exit status of $?"

You'll be able to see that message when running your script in trace mode (set -x / bash -x yourscript), but it'll be otherwise invisible, and because || is branching on curl's exit status, this marks curl as "checked" so set -e won't decide to exit based on its exit status.

_{¹ Not really true: set -e has a bunch of exceptions it doesn't exit over, and those exceptions change between individual shell releases.}

_{² This is a very unintuitively-defined word: For example, when you check the exit status of a function, everything that function calls can become "checked", so set -e's behavior is extremely context-sensitive and hard to predict; what's checked when a function is called one time might not be checked when it's called again later.}

Why does curl not return a value in bash script?

1 Answers1