Pushing data to user browser from server

Question

I want to push data from server to browser. I am already aware of php function ob_flush() which sends output buffer. I need help with a bit of logic. I am using Facebook real time API so i want to push data to user everytime Facebook visits my website.

Here is my code that i am trying to push data to browser but its not working.

<?php
header('Access-Control-Allow-Origin: *');
header('Content-Type: text/event-stream');
ini_set("log_errors", 1);
ini_set("error_log", "php-error.log");
error_log( "LOGS STARTS FROM HERE" );
if(isset($_GET['hub_challenge'])){
    echo $_GET['hub_challenge'];    
}
if($_SERVER['REQUEST_METHOD'] == "POST"){
    $updates = json_decode(file_get_contents("php://input"), true); 
    // Replace with your own code here to handle the update 
    // Note the request must complete within 15 seconds.
    // Otherwise Facebook server will consider it a timeout and 
    // resend the push notification again.
    print_r($updates);
    ob_flush();
    flush();
    //file_put_contents('fb.log', print_r($updates,true), FILE_APPEND);     
    //error_log('updates = ' . print_r($updates, true));              
}
?>

Answer 1

As @som suggested, you can simply use requests with an interval between them, you don't need to use sockets.

But the thing is, you are trying to receive data from the API and pass it right to the browser, all at once. It's best if you separate those two steps.

In the script that receives the data from Facebook, store that data in a database or somewhere else:

if($_SERVER['REQUEST_METHOD'] == "POST"){
    $updates = json_decode(file_get_contents("php://input"), true); 

    insertDataToDatabase($updates); // you'll have to implement this.
}

Then setup a monitoring page:

monitor.php

<script>
lastId = 0;

$(document).ready(function() {
    getNewDataAtInterval();
});

function getNewDataAtInterval() {
    $.ajax({
        dataType: "json",
        url: "getData.php",
        data: {lastId: lastId}
    }).done(function(data) {
        for(i=0; i<data.messages.length; i++) {
            $("#messages").append("<p>" + data.messages[i]['id'] + ": " + data.messages[i]['message'] + "</p>");
            if (data.messages[i]['id'] > lastId) lastId = data.messages[i]['id'];
        }

        setTimeout(getNewDataAtInterval, 10000);
    }).fail(function( jqXHR, textStatus ) {
        alert( "Request failed: " + jqXHR.responseText );
    });
}
</script>

<div id="messages"></div>

At last, create a server-side script to return a JSON with the new messages loaded from the database.

getData.php

$lastId = $_GET['lastId'];
$newMessages = getUpdatesFromDatabase($lastId);

exit(json_encode(array("messages"=>$newMessages)));

function getUpdatesFromDatabase($lastId) {
    // I'm using this array just as an example, so you can see it working.
    $myData = array(
        array("id"=>1,"message"=>"Hi"),
        array("id"=>2,"message"=>"Hello"),
        array("id"=>3,"message"=>"How are you?"),
        array("id"=>4,"message"=>"I'm fine, thanks")
    );

    $newMessages = array();
    foreach($myData as $item) {
        if ($item["id"] > $lastId) {
            $newMessages[] = $item;
            $newLastId = $item["id"];
        }
    }

    return $newMessages;
}

Answer 2

Use of Comet or Prototype will be best practice here. Ajax will increase server load. It will polling the server frequently. Here is some necessary information about using comet.

Here is the example of how to implement comet with php to send real-time data.

Answer 3

With conventional HTTP protocol, web browsers always request your server for response. Once the response is sent, server closes the connection.

True persistent connection technology such as WebSocket is one of the exceptions where a browser makes specific kind of connection to the server and, having the understanding of the intention, the server would maintain the connection open. This way as soon as there's data to be "pushed" to the browser, the connection is always ready. Using this approach, there's no need to save the data temporarily since you'd just "pass it along".

Polling techniques (including long-polling, where the server keeps sending "heartbeat" to the client as if a non-meaningful response is slowly trickling) can be used as work-around, but there's always some interval of time where connection is no longer open, until the next cycle happens. When connection is absent, your only option is to save the data temporarily so when your browser comes back you can push the pending data. In case you're thinking about storing it in a temporary variable, take into account that with PHP script once the execution is completed all data allocated in memory associated in that scope is garbage collected.

Answer 4

I have find Pusher and Redis for push data from server to browser while i was looking for the best solution.

Pusher

You may conveniently consume events broadcast using the Pusher driver using Pusher's JavaScript SDK.

this.pusher = new Pusher('pusher-key');

this.pusherChannel = this.pusher.subscribe('reference_id');

this.pusherChannel.bind('SomeEvent', function(message) {
    console.log(message.user);
});

Redis

If you are using the Redis broadcaster, you will need to write your own Redis pub/sub consumer to receive the messages and broadcast them using the websocket technology of your choice. For example, you may choose to use the popular Socket.io library which is written in Node.

Using the socket.io and ioredis Node libraries, you can quickly write an event broadcaster to publish all events that are broadcast by your application:

var app = require('http').createServer(handler);
var io = require('socket.io')(app);

var Redis = require('ioredis');
var redis = new Redis();

app.listen(6001, function() {
    console.log('Server is running!');
});

function handler(req, res) {
    res.writeHead(200);
    res.end('');
}

io.on('connection', function(socket) {
    //
});

redis.psubscribe('*', function(err, count) {
    //
});

redis.on('pmessage', function(method, channel, message) {
    message = JSON.parse(message);
    io.emit(channel + ':' + message.event, message.data);
});

Answer 5

Why the need to push? Maybe I'm missing a clue here but otherwise is it the only way to solve this problem? You want to be able to set the text of let's say a div we call statusUpdate which shows new statuses from facebook when they are posted? Then you could:

Split up the process into a status collection thread that runs as a daemon that continuously tries to fetch from the FB API (don't know any specs or have any knowledge about FB API but I can only imagine that there are calls to lookup if there are new statuses).

Doesn't matter if the API is streaming or that we need to connect every X second as that we can take into account? I would setup a daemon in php with and then run it with SSH with the command: nohup php daemon.php to start a script with a never ending loop like this:

Define('SLEEP', 1);  // loop every second

while (true) {  

   // do your thing, dont exit

   if( $fbMonkey->checkNewStatus() ){
        $fbMonkey->toDatabase(new_statuses);
  }

   if( some_reason_to_exit() == TRUE ){
      exit;
   }

    sleep(SLEEP);  
}
// While ends with break

Then maybe include into HTML of target user (browser end of the process) a JavaScript function that reads from the table with statuses that the daemon fills and then for the ones that have not been marked as viewed (by users or something similar) and return the unread statuses to the browser. If we create a never ending loop in the browser for it and let it update the div statusUpdate and with the new contents (html or tekst dont matter?). I would let a call like this linger and check every 20 seconds or so and update the div.

http://www.w3schools.com/ajax/tryit.asp?filename=tryajax_xml2

function loadXMLDoc(url)
{
var xmlhttp;
var txt,xx,x,i;
if (window.XMLHttpRequest)
  {// code for IE7+, Firefox, Chrome, Opera, Safari
  xmlhttp=new XMLHttpRequest();
  }
else
  {// code for IE6, IE5
  xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");
  }
xmlhttp.onreadystatechange=function()
  {
  if (xmlhttp.readyState==4 && xmlhttp.status==200)
    {
    txt="<table border='1'><tr><th>Title</th><th>Artist</th></tr>";
    x=xmlhttp.responseXML.documentElement.getElementsByTagName("CD");
    for (i=0;i<x.length;i++)
      {
      txt=txt + "<tr>";
      xx=x[i].getElementsByTagName("TITLE");
        {
        try
          {
          txt=txt + "<td>" + xx[0].firstChild.nodeValue + "</td>";
          }
        catch (er)
          {
          txt=txt + "<td>&nbsp;</td>";
          }
        }
    xx=x[i].getElementsByTagName("ARTIST");
      {
        try
          {
          txt=txt + "<td>" + xx[0].firstChild.nodeValue + "</td>";
          }
        catch (er)
          {
          txt=txt + "<td>&nbsp;</td>";
          }
        }
      txt=txt + "</tr>";
      }
    txt=txt + "</table>";
    document.getElementById('txtCDInfo').innerHTML=txt;
    }
  }
xmlhttp.open("GET",url,true);
xmlhttp.send();
}

Or am I completely of the mark here?

Answer 6

Choice 1 (If you want a reliable system): A combination of queuing server(like AMQP or MQTT etc) and websocket client like http://www.hivemq.com/full-featured-mqtt-client-browser/ will be robust. Almost every language has support for AMQP e.g. in PHP (https://pecl.php.net/package/amqp).

Choice 2 : Use ajax and tell browser to get updates from server in a periodic interval (as already mentioned in comments)

Answer 7

Nice discussion. Me Like:D @andy: great, for me the first time someone could explain the difference real accurate and I get it to:D @Marcos Dimitrio I concur

Myself I run a pretty darn nasty thread pool of Twitter API daemons that do exactly that except for the $_POST push thing from facebook if I understand correctly. It monitors tweets realtime for arrays of hundreds / thousands of keyword cluster via the streamin api Firehose. This is the way to go or suffer terrible defeat otherwise:D IMHO of course. This is the half of two daemons called getTweets and parseTweets.

<?php
ob_start();
require_once('config/phirehose-config.php');
require_once('lib.php');
$oDB = new db;

// run as a daemon aka background process
while (true) {

  // Process all statuses
  $query = 'SELECT cache_id, raw_tweet ' .
    'FROM json_cache';
  $result = $oDB->select($query);
  while($row = mysqli_fetch_assoc($result)) {

    $cache_id = $row['cache_id'];
//    $status = unserialize(base64_decode($row['raw_tweet']));
    $tweet_object = json_decode($row['raw_tweet'],false);


    // JSON payload for statuses stored in the database  
    // serialized base64 raw data

      // Delete cached copy of tweet
      //    $oDB->select("DELETE FROM json_cache WHERE cache_id = $cache_id");

        // Limit tweets to a single language,
        // such as 'en' for English
        //if ($tweet_object->lang <> 'nl') {continue;}

    // Test status update before inserting
    $tweet_id = $tweet_object->id_str;

    if ($oDB->in_table('tweets','tweet_id=' . $tweet_id )) {continue;}

    $tweet_text = $oDB->escape($tweet_object->text);    
    $created_at = $oDB->date($tweet_object->created_at);
    if (isset($tweet_object->geo)) {
      $geo_lat = $tweet_object->geo->coordinates[0];
      $geo_long = $tweet_object->geo->coordinates[1];
    } else {
      $geo_lat = $geo_long = 0;
    } 
    $user_object = $tweet_object->user;
    $user_id = $user_object->id_str;
    $screen_name = $oDB->escape($user_object->screen_name);
    $name = $oDB->escape($user_object->name);
    $profile_image_url = $user_object->profile_image_url;


    // Add a new user row or update an existing one
    $field_values = 'screen_name = "' . $screen_name . '", ' .
      'profile_image_url = "' . $profile_image_url . '", ' .
      'user_id = ' . $user_id . ', ' .
      'name = "' . $name . '", ' .
      'location = "' . $oDB->escape($user_object->location) . '", ' . 
      'url = "' . $user_object->url . '", ' .
      'description = "' . $oDB->escape($user_object->description) . '", ' .
      'created_at = "' . $oDB->date($user_object->created_at) . '", ' .
      'followers_count = ' . $user_object->followers_count . ', ' .
      'friends_count = ' . $user_object->friends_count . ', ' .
      'statuses_count = ' . $user_object->statuses_count . ', ' . 
      'time_zone = "' . $user_object->time_zone . '", ' .
      'last_update = "' . $oDB->date($tweet_object->created_at) . '"' ;     

    if ($oDB->in_table('users','user_id="' . $user_id . '"')) {
      $oDB->update('users',$field_values,'user_id = "' .$user_id . '"');
    } else {            
      $oDB->insert('users',$field_values);
    }

    // percist status to database

    $field_values = 'tweet_id = ' . $tweet_id . ', ' ....


    //... Somethings are to be for da cook alone, its hard work          

            foreach ($entities->hashtags as $hashtag) {

      $where = 'tweet_id=' . $tweet_id . ' ' .
        'AND tag="' . $hashtag->text . '"';     

      if(! $oDB->in_table('tweet_tags',$where)) {

        $field_values = 'tweet_id=' . $tweet_id . ', ' .
          'tag="' . $hashtag->text . '"';   

        $oDB->insert('tweet_tags',$field_values);
      }
    }
    foreach ($entities->urls as $url) {

      if (empty($url->expanded_url)) {
        $url = $url->url;
      } else {
        $url = $url->expanded_url;
      }

      $where = 'tweet_id=' . $tweet_id . ' ' .
        'AND url="' . $url . '"';       

      if(! $oDB->in_table('tweet_urls',$where)) {
        $field_values = 'tweet_id=' . $tweet_id . ', ' .
          'url="' . $url . '"'; 

        $oDB->insert('tweet_urls',$field_values);
      }
    }       
  } 

  if(DEBUG){ 
     echo ob_get_contents();
     ob_clean();
  }else{
     ob_clean();
  }

  // Longer sleep equals lower server load
  sleep(1);
}
?>

Also works great for spiders and crawlers for which a have my own crew to. Show me a better way to do this, all things considerd like resources and scalebility as a persistently connected website widget for FB status updates is really like using Echelon as a tv remote again imho).

Answer 8

If you just need a simple solution and aren't worried about older browser compatibility and you have a low traffic situation, server-sent events could do the trick.

You instantiate it with one line if your push message generation script is on the same server.

var evtSource = new EventSource("messages.php");

Then a function to handle incoming messages.

    evtSource.onmessage = function(e) {

       console.log(e.data);
    }

messages.php needs to have header set as

header("Content-Type: text/event-stream\n\n");

Then do an infinite loop set to an interval as desired.

example:

header("Content-Type: text/event-stream\n\n");
date_default_timezone_set("Pacific/Auckland"); // as required
$sleepTime = 8; // # of seconds delayed between notifications

while (1) 
{   

   // Send a message at $sleepTime second intervals.
   echo 'data: The time is ' . date("H:i:s") . "\n\n";

   ob_end_flush();
   flush();
   sleep($sleepTime);
}

Obviously you'd need to read the messages from somewhere and present as required, but at least this example gives you an idea how to create an event stream without the need to get into the relative complexity of WebSocket.

Disclaimer: I'm not super experienced with PHP but this solution seems to work for me. If there's any issue with this, I'd be keen to hear.

Answer 9

Send a request only when you get the data back from server. And at the server sleep until you find there is something new to send and send the response back.

Do continue the above till your page is alive and you will see the data is pushed. This avoids pinging the server at regular interval.

I am not sure if they are referring this here, https://en.wikipedia.org/wiki/Comet_(programming)