PHP - processing big data

Question

I am trying processing big data by PHP (100 000 000 records). I am downloading every record from different server, then make some text checkings and probably 10% of appropriate records inserting in my DB (MySQL). My problem is:

web browser just finishes and processing is prematurely interrupted
after every xy checkings I want print in browser count of appropriate records, but nothing is printed

MySQL is not problem. It looks like browser can print something after script is finished, but I want interrupt that script (thread) for short time, print my result by browser and then continue. During processing data browser is "freezed". Any ideas?

example:

    for ($i=0; $i<100000000; $i++) {
      if (($i % 1000) == 0) {  //every 1000th iteration
        echo $i;               <=== HERE I need interrupt script and let browser print my result
      }
    }

Browser will only print any thing when it completes excution-- after completing that many loops but on cli you can see that.. — M A SIDDIQUI, Jan 20 '17 at 10:20
Possible duplicate of [PHP Background Processes](http://stackoverflow.com/questions/265073/php-background-processes) — Alessandro Da Rugna, Jan 20 '17 at 10:31
After your `echo $i;` you need `flush();ob_flush();`. I won't post the answer, your design decision to process 100 million records using a browser and PHP is bad. However, you did ask how to output a chunk and this is how you do it. Not by using sleep because that wastes CPU cycles. Browser is "freezed" (frozen) because that's how HTTP works - you send a **request** and you wait for **response**. Your response won't arrive until PHP processes all 100 million records. — Mjh, Jan 20 '17 at 16:38

score 1 · Accepted Answer · edited May 23 '17 at 12:33

First of all, ignore_user_abort is your best friend. Pair with set_time_limit(0) and you've secured your process from dying.

Second, sending something to user is a hard task. The reason being is that whatever you output goes through a couple of buffers. This includes PHP's, Apache's, load balancers that your application might use, browser's, etc. (note: the buffers can usually be easily configured (that is disabled), but they are there for a reason). Therefore simply echoing might not always work.

There is a number of things you can do to address this issue.

One solution is to use real-time communication services like Pusher or Ably (I prefer the latter for nicer free plan) or roll out your own solution using web sockets. Then you would simply send a message every 1k iterations and listen for it in your JS. A bonus is that if user's browser crashes and he reopens it, the process and the updates will be still running correctly. This is (in my opinion) is the most proper way to do this but can be hard to get right.

Another solution would be to split the data. That is JS sends an ajax request, it processes 1k rows, echos and dies. Then JS sends another request which processes the next 2k rows. This can be easier done but relies on the client to send the requests. Also "downloading every record from different server" can be hard to make using this method

PHP - processing big data

1 Answers1