PHP foreach loop to slow for live dashboard. What are my options?

Question

I'm using PHP to GET availability data about 80 Users from a CallCenter. I hope to query the CallCenter for the status/availability of each user every 5 seconds and display the results on a Dashboard.

The CallCenter can only be queried via URL for one user at a time and the username is in the url (so the url is not constant). The answer from the CallCenter is in an XML page. I have been using fopen() to read the content of the url / XML as this is always a random generated XML name and thus the query url is pointed to a directory, not the exact xml url.

I have everything working with a foreach loop, where a unique url is used for USER001 and the content of the XML page is read and a variable is updated. Demo code:

User array:

$users = array( 
        "username" => array("fullname" => "Full Name",
                            "status" => ""  
                            ),
        "username" => array("fullname" => "Full Name",
                            "status" => "")
                );

Foreach Loop:

foreach($users as $username => $userstatus) {
$url = 'http://'.$username.':PasswordForUser@serveraddress.com:80/callcenter/'.$username.'/services/callcenter/';
$get = fopen($url, "r");
    if ($get) {
        while (!feof($get)) {
            $status = fgets($get, 4096);
        }
        fclose($get);
    }
if(strpos($status, 'Available')) { $users[$username]["status"] = "Available"; } else { $users[$username]["status"] = "Offline"; }

}

The issue:
I have a jquery script that refreshes the dashboard every 5 seconds and thus every 5 seconds the CallCenter server is queried of all the Users.

This works, BUT the query actually takes over 1 minute to finish the loop and return all the results to the Dashboard page. It is vital that the CallCenter users can see how many users / agents are available at the given time so they know if enough agents are available to handle the CallCenter load.

Possible solution:
What would be the best solution for my case?

What I have thought of is to create a PHP page for each USER. Have a CRONJOB that runs the pages every x hours (to refresh) and a script on the page that loops through the XML query to the CallCenter every 5 seconds. In the end of each loop (every 5 seconds) the user status is sent to and updates a collector page. Every user page sends the results to the collector page and then the Dashboard page fetshes the data to the collector page every 5 seconds.

Is this a "smart" solution? What would you recommend and how would you code it?
(I´m a PHP begginer)

if you have any control over what happens behind that url, you should try optimizing what happens over there, since it seems to be taking a bit too long. maybe if there're some database queries, optimize those (add indexes, etc.). — , Nov 14 '12 at 13:09
Sadly I have no control over the CallCenter repoting server. That's my main issue. The query takes only 1 seconds for 1 user. Multiply that by 60 :) The PHP doesn´t return the results until all 60 queries are finished. That´s why I thought about having 60 pages to query the server and sending the results to a collector page. — Skuli Axelson, Nov 14 '12 at 13:11
ok, something else then. it seems that you already have the list of users entering the foreach. where do you get that from? i'm asking because maybe in jQuery you could do 1 asynchronous request per user instead of 1 request for all users. that might help your case. — , Nov 14 '12 at 13:17
@SkuliAxelson Try to optimize your script, it seems that file_get_contents() is a little faster http://stackoverflow.com/questions/2749441/php-fastest-way-possible-to-read-contents-of-a-file . And maybe, you could try to create for each user a php file, and launch them simultaneously every X seconds http://stackoverflow.com/questions/6577802/executing-multiple-simultaneous-php-scripts-from-cli — HamZa, Nov 14 '12 at 13:21
The list of users is stored in an manually created array (see example of the array above). The array has the username, encrypted password, fullname and status. NOTE: This is a cross-domain query (I fixed the demo code above). That´s why I used **fopen** and not jquery to read the results from the xml page. — Skuli Axelson, Nov 14 '12 at 13:21
@SkuliAxelson i would create a database, in the the database you put the status of the users. I create separately php files and use a cronjob to update the user status simultaneously (something like status_updater.php). And on the other hand a php file (get_status.php) that parse every 5 seconds the status of the users from the database, of course jQuery is used to get the status clientside. — HamZa, Nov 14 '12 at 13:33
@HamZa DzCyberDeV - Thanks for the file_get_contents() info. I´m going to try as you suggested. Have a DataCollector.php, AgentInfo.php and then one page per 4 users that get´s 4 results and sends them to the DataCollector page. The Dashboard then includes the DataCollector and jquery updates the stats for the users. — Skuli Axelson, Nov 14 '12 at 15:20
@SkuliAxelson a side question does this data change frequently? — ianace, Nov 14 '12 at 15:24
@iance - Yes and No. The CallCenter Agents are on the phone all day long. Whether they are available or not depends on the amount of incoming calls. But as soon as Agent/User is Unavailable it is vital for other agents/users to know that. The other Agents have to be carefull to have at least a few lines open if they are doing something else at work that causes them to be unavailable or away from the phone. — Skuli Axelson, Nov 14 '12 at 15:38

score 1 · Answer 1 · answered Nov 14 '12 at 13:41

1

First use

echo microtime(true)

before and after the GET query to determine the speed of the response you are getting from the Call centre - this will give you an idea if your greatest gains will be from (A) the number of queries or (B) the speed of a single query.

If you can't get the call centre to change their data process you will have too go this route as outlined very quickly below:

Threading your ajax/jQuery calls.

when your page refreshes - execute 4-6 ajax calls passing a thread id to the script, then spilt the users data collection ie: 80 / 4 = 20 users per thread (most browsers can only handle a few http requests, so don't create too many threads as they will basically stack and wait until one is finished - which is not what you want) - once the data is returned you can simple jquery write to an id ie:

<div id="threadresults1"></div>
<div id="threadresults2"></div>
<div id="threadresults3"></div>
...

hope this is clear.

answered Nov 14 '12 at 13:41

on_

556
5
12

http://stackoverflow.com/questions/11235369/measuring-the-elapsed-time-between-code-segments-in-php/11235396 – HamZa Nov 14 '12 at 13:53
Thanks. I used microtime (start and end) to time 1 user query. The result was 0.579805. I´ll look into threading about 4 calls per page. I tried jquery the other day but as this is a cross-domain query I couldn´t get jquery to work. – Skuli Axelson Nov 14 '12 at 15:31
Ah yes that may pose a problem, you can use a php get script to do this for you, the file_get_contents can do both posts and gets to any url. So basically you ajax a local script and the local script gets from the external url, the local ajax call shouldn't delay your timing by much. – on_ Nov 14 '12 at 18:28

ZorleQ · Answer 2 · 2012-11-14T13:51:30.297

I have to completely agree with @Alexandru Mihai in his comment above.

Instead of running the foreach loop in php, drop it to jQuery level, and use your php script only as a 1-to-1 proxy. It will be more resource heavy, but at the same time you will be able to run unlimited number of simultaneous threads / reads, since JS relies on callbacks and php only has to process one name at a time.

So even if your proxy script lags behind, you will still get a valid response for the other operators on js level, so the whole process will feel smooth and responsive.

score 0 · Answer 3 · answered Nov 14 '12 at 13:35

Since we established in the comments that the list of users is statically defined and not queried somehow, you can make the page aware of the list of users.

Then you can query a php page asynchronously for each user: queryPage.php?username=john

In that page you'd only query the callcenter for that specific user. Thus, having 60 asynchronous requests instead of just 1 for all 60 users should help your case.

score 0 · Accepted Answer · answered Nov 27 '12 at 16:15

Just wanted to thank all users that replied. I don´t think there is good solution for my questions with this approach, so I ended with going for as I suggested in the initial post. I created multiple pages that run every 1 minute with a cronjob and collect data. Then other pages fetch the data every x seconds.

PHP foreach loop to slow for live dashboard. What are my options?

4 Answers4