How to fetch next URL from JSON using Bash

Question

I have created a simple script to store response from a third party API

The request is like this..

https://externalservice.io/orders?key=password&records=50&offset=0

The response is as follows:

{
  "results": [
    {
      "engagement": {
        "id": 29090716,
        "portalId": 62515,
        "active": true,
        "createdAt": 1444223400781,
        "lastUpdated": 1444223400781,
        "createdBy": 215482,
        "modifiedBy": 215482,
        "ownerId": 70,
        "type": "NOTE",
        "timestamp": 1444223400781
      },
    },
  ],
  "hasMore": true,
  "offset": 4623406
}

If there is a hasMore attribute, I need to read the offset value to get the next set of records.

Right now I've created a script that simply loops over the estimated number of records (I believed there is) and thought incrementing the offset would work but this is not the case as the offset is not incremental.

#!/bin/bash

for i in {1..100}; do
    curl -s "https://externalservice.io/orders?key=password&records=50&offset=$i" >>outfile.txt 2>&1
done

Can someone explain how I can read continue the script reading the offset value until hasMore=false?

score 1 · Accepted Answer · edited Mar 27 '19 at 21:08

1

You can read a value from a json using the jq utility:

$ jq -r ".hasMore" outfile

true

Here is what you could use:

more="true"
offset=0
while [ $more = "true" ]; do
    echo $offset
    response=$(curl -s "https://example.com/orders?offset=$offset")
    more=$(echo $response | tr '\r\n' ' ' | jq -r ".hasMore")
    offset=$(echo $response | tr '\r\n' ' ' | jq -r ".offset")
done

edited Mar 27 '19 at 21:08

Gilles Quénot

173,512
41
224
223

answered Mar 27 '19 at 17:16

rpld

86
6

Thanks, while this partially solves the question (re: if hasMore=true), the bigger issue is I need to read what the `offset` value. I cannot simply use an incremented value. – InvalidSyntax Mar 27 '19 at 17:19
Sorry, totally missed that part of your question. I updated my answer accordingly – rpld Mar 27 '19 at 17:29
Thanks for the update, I seem to be getting a parsing error: `parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 180, column 8`. I think its the way the bash is formatted, if I use the original snippet you provided it worked. I've tried wrapping `echo $response` like `echo "$response"` but that doesn't work. – InvalidSyntax Mar 27 '19 at 17:35
I've also tried `curl -s 'https://example.com/orders?offset=0' | jq -r '.hasMore'`, this works without any errors. – InvalidSyntax Mar 27 '19 at 17:39
1

Looking at https://stackoverflow.com/questions/52399819/bash-jq-parse-error-invalid-string-control-characters-from-u0000-through-u0?noredirect=1&lq=1, I'd try`echo $response | tr '\r\n' ' ' | jq` – rpld Mar 27 '19 at 17:46
Don't ```echo $response``` json stored in variable, it will lead to all sorts of problems. Main one being it will replace all your ```\n``` sequences within attribute values to actual newlines. Use ```printf "%s\n" "$response"``` instead. Don't remove newlines with ```tr``` - that only covers the error (doesn't fix it) because ```echo``` modifies json to invalid format. – pawrog May 29 '21 at 17:08

score 0 · Answer 2 · answered May 29 '21 at 17:03

You can use jq utility to extract specific attribute from you json response:

$ jq -r ".hasMore" outfile

true

jq expects perfectly valid json input, otherwise it will print error. Most common mistake is to echo json stored in variable. A mistake, because echo will interpret all escaped newlines in your json and will send unescaped newlines within values causing jq to throw parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line message.

To avoid modification to json (and avoiding an error) we need to use printf instead of echo.

Here is the solution without breaking json content:

more="true"
offset=0
while [ $more = "true" ]; do
    echo $offset
    response=$(curl -s "https://example.com/orders?offset=$offset")
    more=$(printf "%s\n" "$response" | jq -r ".hasMore")
    offset=$(printf "%s\n" "$response" | jq -r ".offset")
done

How to fetch next URL from JSON using Bash

2 Answers2