Curl how to scrape over 500 urls - the safe and resource-wise way

Question

I have a list of over 500 urls that i have to scrape because my distributor doesn't offer an api or a csv. The list is actually an array containing the ids of those products that i want to keep track of:

$arr = [1,2,3,...,564];

The url is the same, you only change the id at the end of it:

$url = 'https://shop.com/products.php?id='

Now, on localhost i used a foreach loop to scrape each and everyone of those urls:

foreach($arr as $id){
    
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url . $id);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_MAXREDIRS, 1);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $result = curl_exec($ch);
   
    //preg_meth_all - get the data that i'm looking for
    //put that data into an array

    curl_close($ch);

}

But the problem is that, first of all, i think that that's not wise at all - and i know that for a fact because when i(accidentally) ran the script(on localhost) my access was banned/blocked to that shop.com - getting the message: Too many requests...429:D.

I was trying to sleep that foreach every 10 loops using 10 as modulus

$x = 0;
foreach($arr as $id){
   
    //ch request - get data and add it into an array 
    $x++;
    if($x % 10 == 0){
        sleep(2);
    }      

}

But this takes like forever to execute.

Even tho i am able to connect and take the date that i need from each individual product i want to find a solution using curl(since there's no api nor csv) that will run that script at once but in a safe/wise way.

Is there something like that? If yes, can you please help me understand how?

Thank you!

Answer 1

have a daemon or cronjob that is constantly updating a db 24/7 at safe a safe pace, and whenever you need instant results, just query the db instead of the actual website. if a safe pace is too slow, just keep adding more IP's (use proxys) until it's at an acceptable pace.

Answer 2

UPDATE:

First of all i want to say thank you to all those who answered my question :D

After a few days of reading and trial and error I've reached a conclusion; on which i'm not completely satisfied - so i'll keep trying to search for a better solution but for now this is my result:

First i've added a new column to my table in which i'm saving the time() based on this every time that cron job runs the script i'm selecting the next 30 products which were not updated in the past 12h - i'm running that cron job every 10 minutes. This is the loop:

//get the id's for 30 products that haven't been updated in the past 12 hours
$now = time() - 43200;
$products = $pdo->query("SELECT id FROM products WHERE last_update < $now LIMIT 30");

$bind = [];

foreach($products as $id){
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, 'http://shop.com?product=' . $id[0] . '.html');
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_MAXREDIRS, 1);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $response = curl_exec($ch);

    preg_match_all('!<span class="stock">(.*?)<\/span>!', $response, $data);
    $stock = $data[1];

    array_push($bind, [$stock, time(), $id[0]]);

    curl_close($ch);
    sleep(2);
}

//then i'm just updating this results trough a query
//i'm using PDO do deal with my db

I think that the most important thing here is to sleep that look each and every time this way i didn't get the 429 and actually the execution for 30 products at a time happens pretty fast i mean it takes around 1.5 minutes to complete but i'm avoiding the too many requests thing and it is ran by cron job so i don't really have to do anything.

The limitation with this way of doing things is that by using time() if you have more prodacts that can be fitted in a 12 hours cycle the script will simply start with the first 30 products that haven't been updated in the past 12 hours - but to solve this "problem" i'm thinking about saving a counter in a db table so i can use it to start from X each and every time that script is ran and then update it to X + 30.

Using curl for crawling websites doesn't seem to me as the resource-wise solution but it is one that can take crawling off of your hands.

Again, i'm not 100% satisfied with the way i've written this script but for now it works.

If i'll ever find a better solution i'll post it here.

Thank you!