SSE loads lots of data and paralyzes AJAX post requests

Question

This is my sse_server.php file

include_once 'server_files/init2.php'; //this file includes the connection file to the database and some functions

header('Content-Type: text/event-stream');
header('Cache-Control: no-cache');

$assocArray = array();

$fetch_article = $dbh->prepare("SELECT 
                                        article_author_id, 
                                        article_author_un,
                                        article_id, 
                                        article_cover,
                                        article_title,
                                        article_desc
                                    FROM articles ORDER BY article_id DESC");
$fetch_article->execute();

while ($fetch = $fetch_article->fetch(PDO::FETCH_ASSOC)) 
{
    $article_author_id    = $fetch['article_author_id'];
    $article_author_u     = $fetch['article_author_un'];
    $article_id           = $fetch['article_id']; 
    $article_cover        = $fetch['article_cover'];
    $article_title        = $fetch['article_title'];
    $article_desc         = $fetch['article_desc'];


    $randomNum = rand(0,500);

    //Insert the Random Number along with the article's info | Random Number as a Value and the contents as a Key
                $assocArray[
                            'randomNum'.'|'.                    //0
                            $article_author_id.'|'.             //1
                            $article_author_u.'|'.              //2
                            $article_id.'|'.                    //3
                            $article_cover.'|'.                 //4
                            $article_title.'|'.                 //5
                            $article_desc                       //6

                            ]  = $randomNum;
}

//sort the array
arsort($assocArray, 1); 
//echo '<pre>';
//print_r($assocArray);

//while(true){

    $var = '';
    foreach ($assocArray as $key => $value) {

        $var .=  $value .' => ' . $key . '`|||`<br>';
    }
    echo "retry: 6000\n";
    echo "data: {$var}\n\n";
    ob_flush();
    flush();

//}

and this is how I'm processing the data in client.php file

    <div id="feeds"></div>
    <script>

        if(typeof(EventSource)!=="undefined") {

            var eSource = new EventSource("sse_server.php");


            //detect message received
            eSource.addEventListener('message', function(event) {


            var jsV_feeds = event.data;
            var eventList = document.getElementById("feeds");


            var jsV_feedsArray = jsV_feeds.split('`|||`');  //Seperator



            eventList.innerHTML = jsF_ToFetch(jsV_feedsArray); 

            }, false);
        }
        else {
            document.getElementById("feeds").innerHTML="Whoops! Your browser doesn't receive server-sent events.";
        }


        function jsF_ToFetch(jsP_array) 
        {
            var string = ''; //an empty string 
            for (var i = 0; i < jsP_array.length-1; i++) 
            {

                jsV_Feed = jsP_array[i].split('|');

                jsV_randomNum           = jsV_Feed[0];
                jsV_article_author_id   = jsV_Feed[1];
                jsV_article_author_u    = jsV_Feed[2];
                jsV_article_id          = jsV_Feed[3];
                jsV_article_cover       = jsV_Feed[4];
                jsV_article_title       = jsV_Feed[5];
                jsV_article_desc        = jsV_Feed[6];


                string += jsV_randomNum +'<li><b>'+jsV_article_author_u+'</b><!--process the rest in a similar way--> </li>';


            } // for loop ENDS here
            return '<ol>' + string + '</ol>';
        }

    </script>

The Problem is if I use the foreach loop only, it reconnects every 6 seconds.
And if I wrap the foreach inside a while loop it keeps the connection alive but continously keeps on sending data. This eventually loads up a lot of data within seconds. Also it makes AJAX Post request very slow which is executed via another page simultaneously.

Why is that happening ?
How can I get it to keep the connection open, not send data, and not slow down the AJAX post requests.

PS: I have visited these links -
http://www.html5rocks.com/en/tutorials/eventsource/basics/
PHP Event Source keeps executing

May be I didn't understood them very well enough. If it could be boiled down to even simpler terms, kindly do it!

Thanks in advance!

Answer 1

You want to be using the while(true){} loop that you've commented out in sse_server.php. Your SSE script should never exit (until the socket is closed, which would happen from client-side, i.e. your JavaScript script closing it, or the browser window being closed).

The reason you have problems when using the while loop is that there is no sleep() or wait action inside the while loop. So you are sending data to the client (the same data over and over again!), at maximum rate.

Conceptually, what I'm guessing you are after is this code:

$lastID = 0;
while(true){
  $fetch_article = $dbh->prepare("SELECT something FROM somewhere WHERE conditions AND articleID > ?");
  $results = $fetch_article->execute($lastID);
  if(has 1+ results) {
     foreach($results){
        echo result formatted for SSE
        $lastID = ...;
     }
     flush();ob_flush();
  }
  sleep(5);
}

This is saying it will poll the DB every 5 seconds for new records. If there are no new records it does nothing - just goes back to sleep for another 5 seconds. But if there are new records it pushes them out over SSE to the client.

You can adjust the 5 second sleep to find the balance between CPU usage on the server and latency. Shorter sleep means lower latency (your clients get the new data sooner), but higher CPU on the server.

Aside: The lastID approach above is just some way of detecting what records you have seen, and have not yet seen. It is good if you have a unique ID in your table, which is AUTO_INCREMENT. But, alternatively, if DB records are inserted with a created timestamp, then the query becomes:

$now = 0;
while(true){
    $stmt = prepare( "SELECT ... AND created > ?" );
    $stmt->execute( $now );
    $now = time();
    ...process results ...
    sleep(5);
}

(Slightly safer is to set $now to the maximum created timestamp that was found in results, rather than to time() each time; otherwise it is possible for a record to slip through the cracks and not get sent to clients.)