0

I have a PHP-script that does five different API-request. The script is slow because every API-request "has to wait for the previous to execute". All the API-requests push an array. The scripts echoes out the whole array in the end.

How can I speed up the script using multi processing?

Thanks in advance!

kpmDev
  • 1,330
  • 1
  • 10
  • 28
Secret Coder
  • 70
  • 2
  • 11
  • 2
    Try using [`curl_multi_init()`](http://se2.php.net/manual/en/function.curl-multi-init.php)? I've never used it, but it looks like it could help here. – gen_Eric Apr 03 '14 at 17:05
  • ^, but make sure that the APIs are all from a different host. You might be violating rate limits if you attempt to access an API too quickly. – Dave Chen Apr 03 '14 at 17:06
  • possible duplicate of [How can one use multi threading in PHP applications](http://stackoverflow.com/questions/70855/how-can-one-use-multi-threading-in-php-applications) – kpmDev Apr 03 '14 at 17:09

3 Answers3

1

Multi-threading is certainly an option

<?php
class Fetcher extends Thread {
    public function __construct(Threaded $shared, $page) {
        $this->shared = $shared;
        $this->page   = $page;
    }

    public function run() {
        $data = file_get_contents($this->page);
        if ($data) {
            $this->shared[$this->page] = $data;
        }
    }

    protected $shared;
    protected $page;
}

$start   = microtime(true);
$shared  = new Threaded();
$thread  = 0;
$threads = [];

while ($thread++<8) {
    $threads[$thread] = new Fetcher($shared, sprintf(
        "http://www.google.com/?q=%s", md5(mt_rand())));
    $threads[$thread]->start();
}

foreach ($threads as $thread)
    $thread->join();

printf("Complete in %.3f seconds:\n", microtime(true)-$start);
foreach ($shared as $page => $data) 
    printf("\t%s contained %d bytes\n", $page, strlen($data));
?>

I have used nonsense google searches, but the principles are the same.

Joe Watkins
  • 17,032
  • 5
  • 41
  • 62
-1

Assuming you are calling these APIs over HTTP, you can use multi-curl to run all five API calls simultaneously.

http://php.net/manual/en/function.curl-multi-exec.php

There are several libraries that wrap the PHP functions to make this easier. An example from here: https://github.com/jmathai/php-multi-curl

<?php
  $mc = EpiCurl::getInstance();
  $yahoo = $mc->addURL('http://www.yahoo.com/');
  $google = $mc->addURL('http://www.google.com/');

  echo "The response code for Yahoo! was {$yahoo->code} and Google was {$google->code}";
?>
Brad
  • 159,648
  • 54
  • 349
  • 530
  • The requests are not executed using curl. Simple PHP is used. I do q request and then a grab the result from a JSON array from the "API host". – Secret Coder Apr 03 '14 at 17:08
  • @SecretCoder It sure sounds like you're using HTTP in some way to me... If you're not, this isn't possible. PHP isn't multithreaded, and all of the libraries to hack such functionality in place are similar to multicurl in how they work. – Brad Apr 03 '14 at 17:09
  • http://pastie.org/8991839 This is an example of how I do 2 different API calls. Is you script going to work Brad? – Secret Coder Apr 03 '14 at 17:26
  • @SecretCoder It can, but you would need to rework TwitterAPIExchange. And, you might need to use multi-curl manually since you need more control over the headers. – Brad Apr 03 '14 at 17:30
  • that sounds very complicated... I am searching for something that would separate parst of the script and run them at the same tim. Can I do different files for every API-call and run them att the same time in a new script? – Secret Coder Apr 03 '14 at 17:32
  • @SecretCoder Yes, you could definitely go that route... and call your own scripts with multi-curl. For what it's worth, modifying TwitterAPIExchange probably wouldn't be too difficult. I haven't seen the code, but there's probably a place where you see `curl_exec()`. This effectively gets replaced with `curl_multi_exec()`, but some other modifications would still be needed. If you want to split your current code out into multiple scripts, and call those scripts with my original suggested php-multi-curl code, that would solve your problem in a simple way. – Brad Apr 03 '14 at 17:34
-1

If you are using cURL to fetch data, you should look into https://github.com/petewarden/ParallelCurl which allows you make parallel curl requests.

There is also an example at https://github.com/petewarden/ParallelCurl/blob/master/test.php which is pretty straightforward and explains the usage pretty good.

dkasipovic
  • 5,930
  • 1
  • 19
  • 25