How do I get field from HTTP GET JSON result to file?

Question

I am trying to make a HTTP GET request to an API service and push one of the returned fields in the JSON result to a txt file.

Based on this previously asked question: (Getting JSON value from cURL in Linux Bash) ...I have a bash script as follows...

TOKEN_FILE="/myhome/project/resources/auto_token.txt"

AUTH_RESULT=$(curl -i -H "Content-Type: application/json" "https://access.mywebservice.com/access/oauth/token?grant_type=client_credentials&client_id=123456&client_secret=MySecretPassword");

RESULT_FIELDS=$( cat <<EOF | json_reformat | \
    sed -rne '/:/s@^\s+"(\w+)":\s+"([^"]+)",?@json_\1="\2"@gp'
[$AUTH_RESULT]
EOF
)

if [ -f "$TOKEN_FILE" ]
then 
    echo "$RESULT_FIELDS" > "$TOKEN_FILE"
fi

The expected JSON result looks like this (copied from Postman):

{
  "access_token": "eyJ5bGciOiJSUzI1NiJ6.eyJzY29wZSI6WyJDUl7iLCJNQVAiLCJQVFkiLCJ8R1QiLCJTVFMiLCJUVEwiXSwiaXNzIjoiaHR0cHM6Ly9hY2Nlc3MtdWF0LWFwaS5jb3JlbG9naWMuYXNpYSIsImVudl9hx2Nlc3NfcmVzdHJpY3QiOmZhbHNlLCJleHAiOjE0NjcyODMwODcsImNsaWVudF9pZCI6IjhhOTY4OGJjIn0.F2iQfVsi9zntOxKYrNRukSIwuQ_LGSi_WMIXKII2A3GOEaqs-WmFTi7az9rvvfDsOl9rHy_s_66A6PiCpPftyw21Fl0aZZRoFcKv2H_zDUHuxOEs8V36jHeLghV7pjHwYI_nG68CIGvfuRWFNzQuiMFWc_i8oB3n5noSd8fQqa4",
  "token_type": "bearer",
  "expires_in": 43199,
  "scope": "PROD1 PROD2 PROD3",
  "iss": "https://access.mywebservice.com",
  "env_access_restrict": false
}

I get the following errors returned...

bash-4.1$ ./token_renewal_test_05.sh
: command not foundt_05.sh: line 2:
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
115   576    0   576    0     0   2266      0 --:--:-- --:--:-- --:--:-- 30315
: command not foundt_05.sh: line 3:
: command not foundt_05.sh: line 4:
./token_renewal_test_05.sh: line 14: warning: here-document at line 10 delimited by end-of-file (wanted `EOF')
./token_renewal_test_05.sh: line 13: warning: here-document at line 9 delimited by end-of-file (wanted `EOF')
: command not foundt_05.sh: line 13:
lexical error: invalid char in json text.
                                       sed -rne '/:/s@^\s+"(\w+)":\s+"
                     (right here) ------^
: command not foundt_05.sh: line 10:
./token_renewal_test_05.sh: line 16: syntax error: unexpected end of file

I'm a bit new to bash and despite what appears to be a direct pointer to the issue am having problems resolving this one (note this is version 5)!

Can anyone offer any assistance with this one?

PS: I do not have jq either.

Thanks!

Regards,

Chris

You should get `jq`, or write a small Python/Perl/whatever script that uses a proper JSON parser to process the result. Don't use `sed`. — chepner, Jul 01 '16 at 02:10
I have limited ability to "install" jq properly but have it running in my home dir for now. If I run... `UPDATE=$(curl -i -H "Content-Type: application/json" "https://access.mywebservice.com/access/oauth/token?grant_type=client_credentials&client_id=123456&client_secret=MySecretPassword"); echo $UPDATE ` I get the response: ` Set-Cookie: TS01b4dffd=019ed823624d4ce0a5613954a4e52e4a4e357b9192df4cb403 {"access_token":"eyJhb....MWrD9m2fk","token_type":"bearer","expires_in":43199,"scope":"PROD1 PROD2 PROD3","iss":"https://access.mywebservice.com","env_access_restrict":false} ` — Stan Bridge, Jul 01 '16 at 07:00
When I run the following with jq to to try and parse it... ` AUTH_RESULT=$(curl -i -H "Content-Type: application/json" "https://access.mywebservice.com/access/oauth/token?grant_type=client_credentials&client_id=123456&client_secret=MySecretPassword"); ACCESS_TOKEN=$(./jq -r '.access_token' <<<"$AUTH_RESULT") echo $ACCESS_TOKEN ` ...I get the response... ` parse error: Invalid numeric literal at line 1, column 9 ` — Stan Bridge, Jul 01 '16 at 07:01

score 0 · Accepted Answer · edited May 23 '17 at 12:32

0

Caveat emptor as per this comment on Parsing JSON with UNIX tools.

A working solution for your format:

eval $(cat <<EOF | \
    sed -re 's/(,|\{|\})//g' | \
    sed -re 's/"(\w+)":\s*"?([^"]*)"?$/json_\1='\''\2'\''/'
$JSON
EOF
)

set | grep '^json_'

json_access_token=eyJ5bGciOiJSUzI1NiJ6.eyJzY29wZSI6WyJDUl7iLCJNQVAiLCJQVFkiLCJ8R1QiLCJTVFMiLCJUVEwiXSwiaXNzIjoiaHR0cHM6Ly9hY2Nlc3MtdWF0LWFwaS5jb3JlbG9naWMuYXNpYSIsImVudl9hx2Nlc3NfcmVzdHJpY3QiOmZhbHNlLCJleHAiOjE0NjcyODMwODcsImNsaWVudF9pZCI6IjhhOTY4OGJjIn0.F2iQfVsi9zntOxKYrNRukSIwuQ_LGSi_WMIXKII2A3GOEaqs-WmFTi7az9rvvfDsOl9rHy_s_66A6PiCpPftyw21Fl0aZZRoFcKv2H_zDUHuxOEs8V36jHeLghV7pjHwYI_nG68CIGvfuRWFNzQuiMFWc_i8oB3n5noSd8fQqa4
json_env_access_restrict=false
json_expires_in=43199
json_iss=https://access.mywebservice.com
json_scope='PROD1 PROD2 PROD3'
json_token_type=bearer

edited May 23 '17 at 12:32

Community

1
1

answered Jul 02 '16 at 03:44

srage

990
1
9
27

Thansk @DrewBeres With the following in my bash script... ` eval $(cat < – Stan Bridge Jul 04 '16 at 00:15
Here's the JSON output I'm trying to parse as it is returned when echo'd in bash... `Set-Cookie: TS01b4dffd=019ed82362e63dd7c11ac94b992e5bd506f3bd80494508f1ec5f8dfd {"access_token":"eyJhbGciOiJSUzI1NiJ9.eyJzY29wZSI6WyJDUlQiLCJNQVAiLCJQVFkiLCJTR1QiLCJTVFMiLCJUVEwiXSwiaXNzIjoiaHR0cHM6Ly9hY2Nlc3MtdWF0LWFwaS5jb3JlbG9naYjFzo9M9ukWLFdae1htkc79pdt3o9fJdRU47iGsxPTKz8S5TcJreM6S_4Y3pA3VlYb7HLZKs2WAQLxT7BJ_-2rEvmHqkWWvQi45VlIL7J94wl4jcITw","token_type":"bearer","expires_in":43199,"scope":"PROD1 PROD2 PROD3","iss":"https:access-mywebservice.com","env_access_restrict":false} ` – Stan Bridge Jul 04 '16 at 00:35

score 0 · Answer 2 · answered Jul 04 '16 at 02:05

Thanks again Chepner and Drew

I was having too many issues with Sed (probably due to my lack of exprience). As it turns out, I tried using a lookbehind. Sed doesn't have this but grep does so knowing the strcuture of my JSON response will never chance, I was able to get my token extracted using the following with grep instead... grep -o -P '(?<="access_token":").*(?=","token_type")'

How do I get field from HTTP GET JSON result to file?

2 Answers2