PHP file_get_contents optimization

Question

I use simple file_get_contents feature to grab data from other site and place it on mine.

 <?php
    $mic1link = "https://example.com/yyy.html";
    $mic2link = "https://example.com/zzz.html";
    $mic3link...
    $mic4link...
    $mic5link...
    $mic6link...        
 ?>

 <?php
    $content = file_get_contents($mic1link);
    preg_match('#<span id="our_price_displays" class="price" itemprop="price" content=".*">(.*)</span>#Uis', $content, $mic1);
    $mic1 = $mic1[1];
 ?>
 <?php
    $content = file_get_contents($mic2link);
    preg_match('#<span id="our_price_displays" class="price" itemprop="price" content=".*">(.*)</span>#Uis', $content, $mic2);
    $mic2 = $mic2[1];
 ?>

And fired up by

  <?php echo "$mic1";?> and <?php echo "$mic2";?>

It works but it impacts on performance (delay).

Is there any way to optimize this script or maybe another way to achieve this?

You can either cache them, or use a proper HTTP library like Guzzle to [send the requests in parallel](https://guzzle3.readthedocs.io/batching/batching.html). Which of these makes more sense will depend on your exact use-case. — iainn, Aug 17 '18 at 11:35
i guess using, if they provide, their apis would be the only faster way of doing what you want to do — popeye, Aug 17 '18 at 11:35
https://stackoverflow.com/questions/584826/scrape-web-page-contents — Alive to die - Anant, Aug 17 '18 at 11:36

score 0 · Answer 1 · answered Aug 17 '18 at 12:02

Firstly, as others have said, the first step is to use the Guzzle library for this instead of file_get_contents(). This will help, although ultimately you will always be constrained by the performance of the remote sites.

If at all possible, try to reduce the number of http requests you have to make: Can the remote site aggregate the data from all the requests into a single one? Or are you able to obtain the data via other means? (eg direct requests to a remote database?). The answers here will depend on what the data is and where you're getting it from, but look for ways to acheive this as those requests are going to be a bottleneck to your system no matter what.

If the resources are static (ie they don't change from one request to another), then you should cache them locally and read the local content rather than the remote content on every page load.

Caching can be done either the first time the page loads (in which case that first page load will still have the performance hit, but subsequent loads won't), or done by a separate background task (in which case your page needs to take account of the possibility of the content not being available in the cache if the page is loaded before the task runs). Either way, once the cache is populated, your page loads will be much faster.

If the resources are dynamic then you could still cache them as above, but you'll need to expire the cache more often, depending on how often the data are updated.

Finally, if the resources are specific to the individual page load (eg time-based data, or session- or user-specific) then you'll need to use different tactics to avoid the performance hit. Caching still has its place, but won't be anything like as useful in this scenario.

In this scenario, your best approach is to limit the amount of data being loaded in a single page load. You can do this a number of ways. Maybe by giving the user a tabbed user-interface, where he has to click between tabs to see each bit of data. Each tab would be a different page load, so you'd be splitting the performance hit between multiple pages, and thus less noticable to the user, especially if you've used caching to make it seamless when he flips back to a tab he previously loaded. Alternatively if it all needs to be on the same page, you could use ajax techniques to populated the different bits of data directly into the page. You might even be able to call the remote resources directly from the Javascript in the browser rather than loading them in your back-end php code. This would remove the dog-leg of the data having to go via your server to get to the end user.

Lots to think about there. You'll probably want to mix and match bits of this with other ideas. I hope I've given you some useful tips though.

PHP file_get_contents optimization

1 Answers1