0
<?php

require_once( dirname(__FILE__) . '/shared.php');

function tabelle_holen_hosters( $condition = "" ) {
    global $wpdb;
    $sql = "SELECT * FROM whatever" . $condition;

    if( $table = $wpdb->get_results( $sql, ARRAY_A ) ) {
        return $table;
    } else {
        echo "SQL ERROR in tabelle_holen()";
        var_dump( $table );
        return false;
    }
}

$test = tabelle_holen_hosters( " WHERE working=1 AND alexacheck=0 LIMIT 200" );
?>
<table>
<tbody>
<?php
foreach ( $test as $row ){
    $rank = (int) alexaRank( $row['hoster'] );

    if ($rank == 0)
        $rank = 999999999;

    echo '<tr>';
    echo '<td>' . $row['hoster_id'] . '</td>';
    echo '<td>' . $row['hoster'] . '</td>';
    echo '<td>' . $rank . '<td>';
    echo '</tr>';

    $wpdb->update(
    'fhw_filehosters',
        array(
            'alexa' => $rank,
            'alexacheck' => 1
        ),
        array( 'hoster_id' => $row['hoster_id'] ),
        array(
            '%d',
            '%d'
        ),
        array( '%d' )
    );
    flush(); // <----- NOT working.
}
?>
</tbody>
</table>
<?php
function alexaRank( $url ) {
    $request_url = "http://data.alexa.com/data?cli=10&url=".$url;
    $xml = simplexml_load_file($request_url) or die("feed not loading");
    return $xml->SD->POPULARITY['TEXT'];
}

I have this script, I use the wordpress database class without wordpress. I limited the rows to 200 because else i run into problems like the pageload takes years and maybe php execution time the xml from the alexa functions takes very long. I have a lot of urls to be checked in that database and I want to update them all with one action, meaning removing the 200 limit. What would i have to do to make it like doing 50 at a time, then output and then continue? Ajax? how would I do that?

Update 1:

As i said in comments i am looking for something to make this happen (in the background) of my website, i am actually not really sure if it comes to php execution time limit at this point, but for the "all at once" solution i just tested flush() at the end of the queue and it's not working!

But I am looking for a routine function now I updated the Question, the answers are very generic and I am not that exp. I would love to get see some actually code. If thats too much and tends to be a "write me a program" then I have to figure this out by digging in some time. I will then accept the answer that help best.

James Mitch
  • 519
  • 2
  • 11
  • 26

7 Answers7

2

Your immediate question (how to start output while script is running) is easy to answer:

flush();

But it will not solve your problem, if PHP execution time limit is limiting you. So that leaves you with AJAX:

  • separate the script into two files: one with the empty <table> and JavaScript and one that will get called via AJAX, does the actual processing and returns the table rows.
  • limit the number of processed rows like that: LIMIT $start, 50, where $start is provided via GET/POST parameter
  • in JavaScript, first call the PHP script with start=0, then after processing the response, call it again with start increased by 50. Repeat, until the response is empty.
Fabian Schmengler
  • 24,155
  • 9
  • 79
  • 111
0

If you just want to update the rows and you want to know which items were updated, you don't need a webserver. Instead of 'echo', you could use a logger -> KLogger and write the update information to disk. Using the PHP CLI has advantages in this case. You can set max_execution_time(0) and increase the memory_limit. Additionaly i think it's a bit faster than using apache or nginx. You want to do a bulk update? Take a look at this article. Alternatively you could use MySQL. If there are a lot of different status on working and alexacheck, don't forget to index them.

Community
  • 1
  • 1
chris
  • 1,245
  • 1
  • 10
  • 22
  • In fact, the execution time limit is disabled by default in PHP CLI. – Fabian Schmengler Mar 13 '13 at 11:08
  • This is a desktop app that i have to run from time to time? I am looking for a solution to update them all together yes, but ultimately after the I am looking for some self maintaining routine function that updated the 2700 url's in that table one in a while, something like every someone visits the site update the next 5 rows or so. – James Mitch Mar 13 '13 at 13:56
0

Here's how you should think about it:

  • Create one big mother of god loop.
  • Keep a $total counter and a $temp counter.
  • Iterate through the loop 100 (n) times and increase the counters each time.
  • Once you hit the 100 number limit (or 50 or 200 or whatever) you reset the temp counter.
  • You make the program go to sleep for 3 seconds (or something like that).
  • You iterator through the temp variable n times again, but you request data corresponding to the $total counter.

OR

  • Create one big mother of god loop.
  • Have one counter.
  • Increase the counter in every loop iteration.
  • Once the counter hit's your defined max, you reset it.
  • Sleep.
  • Repeat.

Profit either way.

Now, since you want some code and some explanations:

The PHP manual speaks:

int sleep ( int $seconds )
"Delays the program execution for the given number of seconds."

So the program execution is delayed, you can't reach a limit if you've delayed the execution time.

So you can do this:

$i = 0;
$max = 50;
$sleepTime = 3;

foreach ( $test as $row ){
    $rank = (int) alexaRank( $row['hoster'] );

    if ($rank == 0)
    $rank = 999999999;

    echo '<tr>';
    echo '<td>' . $row['hoster_id'] . '</td>';
    echo '<td>' . $row['hoster'] . '</td>';
    echo '<td>' . $rank . '<td>';
    echo '</tr>';

    $wpdb->update('fhw_filehosters', array(
        'alexa' => $rank,
        'alexacheck' => 1
    ),
    array(
        'hoster_id' => $row['hoster_id']
    ),
    array(
        '%d',
        '%d'
    ),
    array( '%d' )
    );
    ii($i === $max)
    {
        $i = -1;
        sleep($sleepTime);
    }
    $i++;
}

At least try it out, the best way to learn is to experiement.

Jonast92
  • 4,964
  • 1
  • 18
  • 32
  • Like everybody else, very generic and no code. I don't really get the first one. What makes going to sleep and then continue this inside one script helps me? Would that not add up to the possible php execution time limit? – James Mitch Mar 15 '13 at 14:28
  • No, I don't think it will. Just try it out. – Jonast92 Mar 15 '13 at 15:34
  • And of course not, sleeping is not hard, using a counter is not hard, you have a full working code that just needs those two things. Solutions to problems are not always best described in code. The best answers are well explained, no code, algorithms. – Jonast92 Mar 15 '13 at 15:52
0

Long running processes like these might be a candidate for a job queue.

  1. Queue up 50 URLs at a time to a queue.
  2. Have each job queue worker send the results to your main application upon completion.
  3. You application will then handle the display of the results at they come in from the queue.

This should all be done asynchronously.

When the application gets to "maintenance" mode. You can still use the same job queue. Like one of the comments mentioned, you can even have a cron job which sends jobs to the queue twice a day.

rexposadas
  • 3,109
  • 3
  • 27
  • 49
  • From the wiki: a job queue was introduced to perform long-running tasks asynchronously. Take a look at http://en.wikipedia.org/wiki/Help:Job_queue. – rexposadas Mar 17 '13 at 05:57
  • ähm by looking shortly at this link it seems to be a mediawiki feature? How does this suppose to help me with my script? – James Mitch Mar 19 '13 at 06:56
0

I usually use function ob_start and ob_flush. It has worked for me.

georgecj11
  • 1,600
  • 15
  • 22
0

first of all will be great to split working wit base and front - end code so :

  1. ajax.getdata.php //Here we getting the data according to parameter given

    $page = $_GET["page"]; // I skip the validation, you should do it be yourself

require_once( dirname(__FILE__) . '/shared.php');

function tabelle_holen_hosters( $condition = "" ) {
    global $wpdb;
    $sql = "SELECT * FROM whatever" . $condition;

    if( $table = $wpdb->get_results( $sql, ARRAY_A ) ) {
        return $table;
    } else {
        echo "SQL ERROR in tabelle_holen()";
        var_dump( $table );
        return false;
    }
}

function alexaRank( $url ) {

    $request_url = "http://data.alexa.com/data?cli=10&url=".$url;
    $xml = simplexml_load_file($request_url) or die("feed not loading");
    return $xml->SD->POPULARITY['TEXT'];
}

$limit = 200;

$start= $page*$limit;

$test = tabelle_holen_hosters( " WHERE working=1 AND alexacheck=0 LIMIT $start,$limit" );



 $str = '<table>';

foreach ( $test as $row ){

    $rank = (int) alexaRank( $row['hoster'] );

    if ($rank == 0)
        $rank = 999999999;

    $str.= '<tr>';
    $str.= '<td>' . $row['hoster_id'] . '</td>';
    $str.= '<td>' . $row['hoster'] . '</td>';
    $str.= '<td>' . $rank . '<td>';
    $str.= '</tr>';

    $wpdb->update(
    'fhw_filehosters',
        array(
            'alexa' => $rank,
            'alexacheck' => 1
        ),
        array( 'hoster_id' => $row['hoster_id'] ),
        array(
            '%d',
            '%d'
        ),
        array( '%d' )
    );

}

$str.='</table>';

echo $str;

exit;

?>

  2. index.html // building front-end view , and sending ajax request to server.

    <div id="content">
</div>

<div id="controls">
<span onCLick="goPrev()">
Prev
</span>
<span onCLick="goNext()">
Next
</span>
</div>

<script>
var page=0;

//this functions can be moved to separate file

function goNext(){
page++;
//here should be some limit for max page
readData();
}

function goPrev(){
if(page==0) return false;
page--;
readData();
}

function readData(){
$.ajax({
"url":"/path/to/ajax.getdata.php?page="+page,
}).done(function(data){
$("#content").html(data);
});
}

readData();
</script>

</body>
</html>

so now you can define be yourself how many record in one time you will show, and can go to next or previous page by controls click.

Alex Zheka
  • 566
  • 2
  • 11
  • Nice, that looks like something i taught about, I am very busy atm and not tested it, but before some automatically rewarding takes place I accept your answer. Looks most detailed. Only 18 hours left. – James Mitch Mar 19 '13 at 18:53
  • It's not a full answer - you will need to add some verifications and do code more structured, but it is the way how you can do what you need. I'm glad that my answer was helpful. – Alex Zheka Mar 20 '13 at 07:13
0

Another way you could think: Is there only certain rows that I could update for a single person visiting the site? To get the illusion it is all being updated.

In online browser games this is the case a lot. They give the illusion everything is up-to-date but really it is just that players information that has been updated.

Alternatively you could see if your server host has an option for 'scheduled scripting', I know a few that allow this. It would trigger a script to run at intervals or certain times of the day.

I hope this helped in someway.

William James
  • 133
  • 1
  • 1
  • 10