PHP Foreach Loop Speed

Question

I'm trying to take a list of 40,000 items and run a json request for each item. The process is super slow which I suspect is because each json request is finishing before the next request starts.

$result = mysql_query("SELECT * FROM $tableName");

while($row = mysql_fetch_array($result))
{
 checkRank($row['Domain']);
}

checkRank is running something to the effect of:

$json = file_get_contents($jsonurl,0,null,null);

I'm thinking I could run 10 checkRanks at a time to speed the process up? Any other ideas / suggestions?

UPDATE:

For example this loop runs through my array in 27 secs.

for ($i=0; $i<=100; $i++) {
checkRank($domains[$i]);
$i++;
checkRank($domains[$i]);
$i++;
checkRank($domains[$i]);
$i++;
checkRank($domains[$i]);
$i++;
checkRank($domains[$i]);
$i++;
echo "checking " . $i . "<br/>";

}

The loop below takes over 40 seconds with the same array.

for ($i=0; $i<=100; $i++) {
checkRank($domains[$i]);
echo "checking " . $i . "<br/>";

}

Is $jsonurl under your control or is it an external service? — Alex Howansky, Sep 11 '12 at 19:29
http://stackoverflow.com/questions/124462/asynchronous-php-calls — Brendan Long, Sep 11 '12 at 19:30
@user1647347: Your code is vulnerable and not safe. Please read my recent post about SQL injections, you could find it interesting: http://stackoverflow.com/questions/11939226/sql-injections-and-adodb-library-general-php-website-security-with-examples — Ilia Ross, Sep 11 '12 at 19:32
$tableName is not a post or get variable..its listed in my config.php and gets included..I don't think sql injection is an issue here — user1647347, Sep 11 '12 at 19:37
If you're pulling that much data from the database at once then of course it's going to be slow! Use a WHERE clause on your query to restrict the resultset to just the records you're interested in. You'll save memory, reduce database traffic and reduce the number of loops you'll need to execute. — GordonM, Sep 11 '12 at 21:23

sarsnake · Answer 1 · 2012-09-11T19:48:29.423

1

Unless there is something else that you haven't mentioned, the best way to do this would be to do one JSON request and pass your items to it, and get the equal number of results back. That way, you minimize the server response time. I am not sure if you want to send 40,000 items in though, you might want to divide it into 2 parts, but that you can test this later.

so you checkRank() would look something like

function  checkRank($domainsArray) {
$json = file_get_contents($jsonurl,$domainsArray);
}

edited Sep 11 '12 at 19:48

answered Sep 11 '12 at 19:31

sarsnake

26,667
58
180
286

Not sure exactly what you mean? Isn't that what I'm doing now? Can you elaborate please? – user1647347 Sep 11 '12 at 19:38
Right now you are calling a web service (or a server page) 40,000 times which makes your call very slow. I would suggest modifying your server page that you call using JSON to return multiple results, not one. – sarsnake Sep 11 '12 at 19:45
I dont have control over that. The jsonurl is compete.com API I wrote the same script using ajax and jquery and it was much faster because I performed 10 ajax requests at once. I was hoping to be able to do the same thing here. – user1647347 Sep 11 '12 at 19:47
well 10 requests is not 40,000. Consider changing it since your scenario changed quantitatively speaking. – sarsnake Sep 11 '12 at 19:59
no, I'm not saying make 10 requests. Make 10 at a time instead of 1 at a time. – user1647347 Sep 11 '12 at 20:23
in reality with the way your code is, you're still making 40,000 ajax requests, which means you are going back and forth between server and client 40,000 times, which will be slow. Your choices are: caching, reducing # of requests (ideally one instead of 40,000) or multithreading. The easiest option is to reduce the number of ajax requests. – sarsnake Sep 11 '12 at 20:31

score 1 · Answer 2 · edited May 23 '17 at 12:12

1

Not sure if this will help much because I don't work with PHP but I did find this.

How can one use multi threading in PHP applications

edited May 23 '17 at 12:12

Community

1
1

answered Sep 11 '12 at 19:32

Ccorock

892
12
37

Looks interesting, I'll have to test it out. – user1647347 Sep 11 '12 at 19:41

score 1 · Accepted Answer · answered Sep 11 '12 at 21:54

1

http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/

This seems to be a nice way to speed up the processing. Thanks for all the input guys.

answered Sep 11 '12 at 21:54

user1647347

507
1
8
23

score 1 · Answer 4 · answered Sep 13 '12 at 18:30

1

... it's a shame, I read on Stack Overflow today that PHP could not support threading as it would require fundamental changes to the language ...

https://github.com/krakjoe/pthreads

answered Sep 13 '12 at 18:30

Joe Watkins

17,032
5
41
62

PHP Foreach Loop Speed

4 Answers4