Get large data from API with pagination

Question

I'm trying to GET a large amount of data from the API (over 300k records). It has pagination (25 records per page) and request limit is 50 request per 3 minutes. I'm using PHP curl to get the data. The API needs JWT token authorization. I can get a single page and put its records into an array.

...
$response = curl_exec($curl);
curl_close($curl);
$result = json_decode($response, true);

The problem is I need to get all records from all pages and save it into array or file. How to do it? Maybe I should use JS to do it better?

Best regards and thank you.

Answer 1

Ideally use cron and some form of storage, database or a file.

It is important that you ensure a new call to the script doesn't start unless the previous one has finished, otherwise they start stacking up and after a few you will start having server overload, failed scripts and it gets messy.

1. Store a value to say the script is starting.

2. Run the CURL request.

3. Once curl has been returned and data is processed and stored change the value you stored at the beginning to say the script has finished.

4. Run this script as a cron in the intervals you deem necessary.

A simplified example:

    <?php 

    if ($script_is_busy == 1) exit();

    $script_is_busy = 1;

    // YOUR CURL REQUEST AND PROCESSING HERE
   
    $script_is_busy = 0;

    ?>

Answer 2

Okay. I misinterpreted what you needed. I have more questions.

Can you do one request and get your 50 records immediately? That is assuming when you said 50 requests per 3 minutes you meant 50 records.
Why do you think there is this 50/3 limitation?
Can you provide a link to this service?
Is that 50 records per IP address?
Is leasing 5 or 6 IP addresses an option?
Do you pay for each record?
How many records does this service have total?
Do the records have a time limit on their viability.

I am thinking if you can use 6 IP addresses (or 6 processes) you can run the 6 requests simultaneously using stream_socket_client().
stream_socket_client allows you to make simultaneous requests.
You then create a loop that monitors each socket for a response.
About 10 years ago I made an app that evaluated web page quality. I ran

• W3C Markup Validation
• W3C CSS Validation
• W3C Mobile OK
• WebPageTest
• My own performance test.

I put all the URLs in an array like this:

   $urls = array();
   $path = $url;
   $url = urlencode("$url");
   $urls[] = array('host' => "jigsaw.w3.org",'path' => "/css-validator/validator?uri=$url&profile=css3&usermedium=all&warning=no&lang=en&output=text");
   $urls[] = array('host' => "validator.w3.org",'path' => "/check?uri=$url&charset=%28detect+automatically%29&doctype=Inline&group=0&output=json");
   $urls[] = array('host' => "validator.w3.org",'path' => "/check?uri=$url&charset=%28detect+automatically%29&doctype=XHTML+Basic+1.1&group=0&output=json");  

Then I'd make the sockets.

  foreach($urls as $path){
    $host = $path['host'];
    $path = $path['path'];
    $http = "GET $path HTTP/1.0\r\nHost: $host\r\n\r\n";
    $stream = stream_socket_client("$host:80", $errno,$errstr, 120,STREAM_CLIENT_ASYNC_CONNECT|STREAM_CLIENT_CONNECT); 
    if ($stream) {
      $sockets[] = $stream;  // supports multiple sockets
      $start[] = microtime(true);
      fwrite($stream, $http);
    }
    else { 
      $err .=  "$id Failed<br>\n";
    }
  }

Then I monitored the sockets and retrieved the response from each socket.

while (count($sockets)) {
  $read = $sockets; 
  stream_select($read, $write = NULL, $except = NULL, $timeout);
  if (count($read)) {
    foreach ($read as $r) { 
      $id = array_search($r, $sockets); 
      $data = fread($r, $buffer_size); 
      if (strlen($data) == 0) { 
     //   echo "$id Closed: " . date('h:i:s') . "\n\n\n";
        $closed[$id] = microtime(true);
        fclose($r); 
        unset($sockets[$id]);
      } 
      else {
        $result[$id] .= $data; 
      }
    }
  }
  else { 
 //   echo 'Timeout: ' . date('h:i:s') . "\n\n\n";
    break;
  }
}  

I used it for years and it never failed.
It would be easy to gather the records and paginate them.
After all sockets are closed you can gather the pages and send them to your user.

Do you think the above is viable?

---

---

JS is not better.

Or did you mean 50 records each 3 minutes?

This is how I would do the pagination.
I'd organize the response into pages of 25 records per page.
In the query results while loop I'd do this:

$cnt = 0;
$page = 0;
while(...){
    $cnt++
    $response[$page][] = $record;
    if($cnt > 24){$page++, $cnt = 0;}
}
header('Content-Type: application/json');
echo json_encode($response);

Answer 3

I would use a series of requests. A typical request takes at most 2 seconds to fulfill, so 50 requests per 3oo secs does not require parallel requests. Still you need to measure time and wait if you don't want to be banned for DoS. Note that even with parallelism, curl supports it as far as I remember. When you reach the request limit you must use the sleep function to wait until you can send new requests. For PHP the real problem that it is a long running job, so you need to change settings, otherwise it will timeout. You can do it this way: Best way to manage long-running php script? As of nodejs, I think it is a lot better solution for this kind of async tasks, because the required features come naturally with nodejs without extensions and such things, though I am biased towards it.