Event Source -> Server returns event stream in bulk rather then returning in chunk

Question

I have a php script that import large data from csv files with validations.
For that I need to show progress to the user. I have used Event Streaming for that.
When I echo something, I want it to be transferred to client one by one instead of server sent whole output in bulk.
I had already played around with ob_start(), ob_implicit_flush() & ob_flush(), but they didn't work.
My script is working perfect on another server. Below server configurations are given:

Server configuration on which the code is not responding as desired, i.e.

OS: Linux
PHP Version 5.4.36-0+deb7u3
Server API: CGI/FastCGI 
Memory_limit: 128M
output_buffering: no value

As I have said, the code is working properly on another server which has the almost same configuration, i.e.

OS: Linux
PHP Version 5.4.37
Server API: CGI/FastCGI 
Memory_limit: 256MB
output_buffering: no value

Below is my sample code for sending event:

<?php
header("Content-Type: text/event-stream");
header("Cache-Control: no-cache");
header("Access-Control-Allow-Origin: *");

$lastEventId = floatval(isset($_SERVER["HTTP_LAST_EVENT_ID"]) ? $_SERVER["HTTP_LAST_EVENT_ID"] : 0);
if ($lastEventId == 0) {
    $lastEventId = floatval(isset($_GET["lastEventId"]) ? $_GET["lastEventId"] : 0);
}

echo ":" . str_repeat(" ", 2048) . "\n"; // 2 kB padding for IE
echo "retry: 2000\n";

// event-stream
$i = $lastEventId;

while ($i <= 100) {
    if($i==100){
        echo "data: stop\n";
        ob_flush();
        flush();
        break;
    } else {
        echo "id: " . $i . "\n";
        echo "data: " . $i . ";\n\n";
        ob_flush();
        flush();
        sleep(1);
    }
    $i++;
}
?>

Below is my client page on which I need response:

<!DOCTYPE html>
<html>

<head>
    <meta charset="utf-8" />
    <title>EventSource example</title>
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <script src="../jquery/eventsource.js"></script>
    <script>
        var es = new EventSource("events.php");
        var listener = function(event) {
            console.log(event.data);
            var type = event.type;
            if (event.data == 'stop') {
                es.close();
            } else {
                var div = document.createElement("div");
                div.appendChild(document.createTextNode(type + ": " + (type === "message" ? event.data : es.url)));
                document.body.appendChild(div);
            }
        };
        var errlistener = function(event) {
            es.close();
        }
        es.addEventListener("open", listener);
        es.addEventListener("message", listener);
        es.addEventListener("error", errlistener);
    </script>
</head>

<body>
</body>

</html>

you mean your live http strimming is working on other server ???? — HaRsH, Feb 19 '15 at 13:35
Yes, it is working on another server... but not with the new one — ba1ar, Feb 19 '15 at 13:42

Barkermn01 · Answer 1 · 2018-09-13T16:06:22.600

Your best method to return chucked data to the browser is to use Web Sockets get the client to open a socket to your file reader then you can chunk the data to the browser without a problem.

Then once it has finished you can close the socket.

~~a good tutorial for web sockets http://www.phpbuilder.com/articles/application-architecture/optimization/creating-real-time-applications-with-php-and-websockets.html~~

with this method you can then if you wanted implement verification so the server is not just sending chunks it's sends the chunks on request by javascript

So your Client could say i need chunk 5 and your server implement something like

$requestedChunk = 5; // this would be set by the javascript sending the request
$chunkSize = 256; // this would be your chunk size;

$readPossition = $requestedChunk * $chunkSize;

Link no longer works so here is one built on Ratchet: https://blog.samuelattard.com/the-tutorial-for-php-websockets-that-i-wish-had-existed/

it's actually a 403 just showing as 404 looks at the address you are directed to but use google "php WebSocket server example" and you will find plenty — Barkermn01, Sep 13 '18 at 15:33

score 0 · Answer 2 · answered Nov 27 '15 at 16:57

I had a similar problem. Event streams were working as expected (returning chunks) on a server using the Apache 2.0 Handler but not on a server using FastCGI (returning it in bulk). I assumed that something in FastCGI is the culprit and so tried to resolve the problem by switching to CGI. Now the event stream works as expected.

Whether using CGI or FastCGI the Server API shows up as CGI/FastCGI so I assume that the server it works on for you is running CGI and the server it doesn't work on for you is running FastCGI. Try changing the non-working server to CGI.

As for why it doesn't work in FastCGI I'm not entirely sure but unless it's a necessary requirement and CGI isn't possible then the above solution should work.

it's not just FastCGI / even mod_php had this problem – Barkermn01 Dec 01 '15 at 11:33 — Barkermn01, Dec 01 '15 at 11:33

score 0 · Answer 3 · answered Dec 04 '15 at 15:16

Many things can prevent chunked response such as but not limited to;

Proxy or any other buffering mechanism on web server
When "Output buffering" is "on" in php.ini (you should explictly set it to off)
When gzip is enabled on web server

You should check these at first.

Event Source -> Server returns event stream in bulk rather then returning in chunk

3 Answers3

Linked