Loop through paginated data JSON using PHP

Question

I'm working with a REST API that returns data in paginated format. 1 page will have 100 records. If there is more data there will be a "hasMore" parameter and "offset" parameter defined which you can use to retrieve the next page of results.

So far I've got the following code:

function getData(){
    $curl = curl_init();
    curl_setopt_array($curl, array(
      CURLOPT_URL => "ENDPOINT",
      CURLOPT_RETURNTRANSFER => true,
      CURLOPT_ENCODING => "",
      CURLOPT_MAXREDIRS => 10,
      CURLOPT_TIMEOUT => 30,
      CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
      CURLOPT_CUSTOMREQUEST => "GET",
      CURLOPT_HTTPHEADER => array(
        "cache-control: no-cache"
      ),
    ));
    $response = curl_exec($curl);
    $data = json_decode($response);

    //if there is more get them.....
    if($data->hasMore == 1){ //make another request...
        while($data->hasMore == 1){ 
            $offset = $data->offset;
            $curl = curl_init();
            curl_setopt_array($curl, array(
              CURLOPT_URL => "ENDPOINT",
              CURLOPT_RETURNTRANSFER => true,
              CURLOPT_ENCODING => "",
              CURLOPT_MAXREDIRS => 10,
              CURLOPT_TIMEOUT => 30,
              CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
              CURLOPT_CUSTOMREQUEST => "GET",
              CURLOPT_HTTPHEADER => array(
                "cache-control: no-cache"
              ),
            ));
            $response = curl_exec($curl);
            $data = json_decode($response);
            $allData = array_merge($data->data, $data->data);
        }
    }
    return $data; //return the results for use
}

My issue is that I'm struggling to have it check if there is more data and if there is request the next page all within the same function. I hope this makes sense and I'd appreciate any help or advice you can offer.

Right now I have it working as I know there is another page. The issue is how can I create it so that it will check the parameter to see if there is more and if there is another page request and append to the existing array of data.

How is the API endpoint expecting `offset` to be passed to it? As a query string param (like endpoint?offset=x) or as part of a POST body? Your example code doesn't actually use `$offset` in the second curl request. — Anthony, Jul 16 '18 at 23:41
I'm passing the offset parameter as part of the query string in the second Curl request. — Javacadabra, Jul 16 '18 at 23:53

Anthony · Accepted Answer · 2018-07-16T23:52:30.597

Based on @TommyLee answer, but using the returned offset (just guessing how it is supposed to be used):

function getData($offset = 0){
    $curl = curl_init();
    curl_setopt_array($curl, array(
      CURLOPT_URL => "ENDPOINT" . "?offset=" . $offset,
      CURLOPT_RETURNTRANSFER => true,
      CURLOPT_ENCODING => "",
      CURLOPT_MAXREDIRS => 10,
      CURLOPT_TIMEOUT => 30,
      CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
      CURLOPT_CUSTOMREQUEST => "GET",
      CURLOPT_HTTPHEADER => array(
        "cache-control: no-cache"
      ),
    ));
    $response = curl_exec($curl);
    $data = json_decode($response);

    return $data; //return the results for use
}

$allData = array();

do {
    $offset = $data->offset ?: 0;
    $data = getData($offset);

    foreach($data->data as $row){
         $allData[] = $row;
    }
} while($data->hasMore == 1);

This way, if nothing is passed to getData(), it assumes the offset is 0, otherwise, you pass it the next offset from the last response.

Again, how the offset is used isn't clear, so you might need to adjust my example to fit the proper usage for $offset.

I modified this and self-posted the answer to what seems to be a similar question - https://stackoverflow.com/a/54238186/1375163 — Robert Andrews, Jan 17 '19 at 14:36

score 1 · Answer 2 · answered Jul 16 '18 at 23:33

Your function is fine, but you should really separate the getData function and looping part.

function getData(){
    $curl = curl_init();
    curl_setopt_array($curl, array(
      CURLOPT_URL => "ENDPOINT",
      CURLOPT_RETURNTRANSFER => true,
      CURLOPT_ENCODING => "",
      CURLOPT_MAXREDIRS => 10,
      CURLOPT_TIMEOUT => 30,
      CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
      CURLOPT_CUSTOMREQUEST => "GET",
      CURLOPT_HTTPHEADER => array(
        "cache-control: no-cache"
      ),
    ));
    $response = curl_exec($curl);
    $data = json_decode($response);

    return $data; //return the results for use
}

do{
    $data = getData();
    $i = 0;
    for(i=0; i<data.length; i++){
         //do data processing here
    }
}while(data->hasMore == 1);

This is good, but it doesn't use the 'offset' parameter mentioned by the OP (neither does the OP's example code, but I think that's part of the problem). Presumably this offset needs to be passed to `getData()` so that it gets the next page/batch of data. — Anthony, Jul 16 '18 at 23:39

Loop through paginated data JSON using PHP

2 Answers2

Linked