What is the best way of showing progress on an Ajax call?

Question

I have an Ajax call that updates 5,000 records in a database so this takes a lot of time. I have an Ajax "Loading image" showing that something is happening, but I am looking for a better way to show "Updating 50 of 5000 . . .", "Updating 200 of 5000", or something like that.

What is the best way to do something like this in Ajax / jQuery without doing 5000 different posts?

Answer 1

The best I think is using Comet.

In Comet-style applications, the server can essentially push data to the client (instead of the client request data from the server again and again.). The client needs to only connect to server once. and then server will keep pushing data back to client.

From Wikipedia:

Comet is a programming technique that enables web servers to send data to the client without having any need for the client to request it. It allows creation of event-driven web applications which are hosted in the browser.

And now let's see how Comet works. See the following server-side code. here a while loop is being used, you can instead set your own condition. In the while loop, the page writes a datetime and flushes and then sleeps for 1/2 seconds.

ASP.NET page code-behind: Service.aspx.cs

public static string Delimiter = "|";

protected void Page_Load(object sender, EventArgs e)
{
    Response.Buffer = false;

    while (true)
    {
        Response.Write(Delimiter
            + DateTime.Now.ToString("HH:mm:ss.FFF"));
        Response.Flush();

        // Suspend the thread for 1/2 a second
        System.Threading.Thread.Sleep(500);
    }

    // Yes I know we'll never get here,
    // it's just hard not to include it!
    Response.End();
}

Client side code - pure JavaScript

Only make the request once, and then keep checking the data in the readyState === 3 of XMLHttpRequest.

function getData()
{
    loadXMLDoc("Service.aspx");
}

var req = false;
function createRequest() {
    req = new XMLHttpRequest(); // http://msdn.microsoft.com/en-us/library/ms535874%28v=vs.85%29.aspx
}

function loadXMLDoc(url) {
    try {
        if (req) { req.abort(); req = false; }

        createRequest();

        if (req) {
            req.onreadystatechange = processReqChange;
            req.open("GET", url, true);
            req.send("");
        }
        else { alert('unable to create request'); }
    }
    catch (e) { alert(e.message); }
}

function processReqChange() {
    if (req.readyState == 3) {
        try {
            ProcessInput(req.responseText);

            // At some (artibrary) length   recycle the connection
            if (req.responseText.length > 3000) { lastDelimiterPosition = -1; getData(); }
        }
        catch (e) { alert(e.message); }
    }
}

var lastDelimiterPosition = -1;
function ProcessInput(input) {
    // Make a copy of the input
    var text = input;

    // Search for the last instance of the delimiter
    var nextDelimiter = text.indexOf('|', lastDelimiterPosition + 1);
    if (nextDelimiter != -1) {

        // Pull out the latest message
        var timeStamp = text.substring(nextDelimiter + 1);
        if (timeStamp.length > 0) {
            lastDelimiterPosition = nextDelimiter;
            document.getElementById('outputZone').innerHTML = timeStamp;
        }
    }
}

window.onload = function () { getData(); };

Reference

Answer 2

I would let the function that is doing the big update record in a SESSION variable its current progress after each single (or so many) update, and use a separate AJAX script to retrieve this progress value from the SESSION and let JavaScript use this to update your progress bar/text.

Answer 3

I am assuming you are currently using one POST for all records in the batch update and placing the loading image between call and return.

Rather than having the server wait to return until completing the update, have it return immediately with a special ID for that batch update. Then implement a server call that returns the status of the batch update, which your progress dialog can call to report the progress.

var progressCallback = function(data){
  //update progress dialog with data
  //if data does not indicate completion
    //ajax call status function with "progressCallback" as callback
});
//ajax call status function with "progressCallback" as callback

Answer 4

I would fire an Ajax callback once every n milliseconds that can query on how much is done (e.g. number of records updated) and use that to display a progress bar. Something like the way this works.

Answer 5

You could update the response buffer with a progress, flushing your response buffer periodically from the server.

But you may have trouble reading a request before it's complete via xhttpr. You might be able to make your request via an iframe, and have that load in progress via 'http streaming'.

But even that can be sketchy. HTTP is not the meant to transfer things piecemeal/fragmented. Like others point out, it would be best to make a separate subsequent calls to get the status of the operation.

Answer 6

I just got an idea while reading the replies.

JavaScript and PHP share cookies,

1. Create a cookie with JavaScript when an Ajax call is made.
2. In an Ajax PHP file increment the value in that cookie with each SQL update.
3. In JavaScript a recursive function will read that particular cookie and will update progress bar numbers.

Advantages:

1. Only 1 Ajax call.
2. Less load on the server.

Answer 7

I am assuming that you have a reason for iterating through each record individually, instead of simply running a SQL statement.

If that is the case, simply make an ajax call every 200 or so iterations. If you do it for each group of 200 records, it will only consume 50 Ajax calls.

Something like (pseudocode):

If iterationNumber mod 200 == 0
    // Make Ajax call.

Answer 8

I'm not sure what the server-side is where you are posting to, but you should be able to apply this method to most programming languages. I'm using PHP as an example.

On the HTML-side, have some function that updates the progress bar to a given width. I'm calling this function 'setProgress' for this example, which takes a number to update the progress bar to.

In the server-side code, do chunks of updates (say 100 per iteration) and generate output. By outputting a javascript call for each iteration:

<?php
  while () { // Whatever your loop needs to be.
  // Do your chunk of updates here.
?>
   <script type="text/javascript">
     setProgress(100);
   </script>
<?php
    flush(); // Send what we have so far to the browser so the script is executed.
  }
  setProgress(5000); // All done!
?>

After echoing this, flush the output buffer to make sure this data is sent to the browser. Because it is a complete script tag, the browser will execute the javascript inside, updating the progress bar to whatever value you passed to it.

To get the whole thing working you will need to add some calculations to make sense of the numbers you are calling to the progress bar, but I'm assuming that shouldn't be too much of a problem. It would probably make more sense to have setProgress in the example use percentages, though I wanted to stay clear of the calculations needed for the sake of clarity.

Answer 9

Create a simple table like this:

CREATE TABLE progress_data (
  statusId int(4) NOT NULL AUTO_INCREMENT,
  progress float DEFAULT NULL COMMENT 'percentage',
  PRIMARY KEY (id_progress_data)
);

JQuery code:

//this uses Jquery Timers http://plugins.jquery.com/project/timers
$('#bUpdate').click(function() {
    //first obtain a unique ID of this operation - this has to by synchronized
    $.ajaxSetup({'async': false});
    $.post('ajax.php', {'operation': 'beginOperation'}, function(data) {
        statusId = parseInt(data.statusId);
    });
    //now run the long-running task with the operation ID and other params as necessary
    $.ajaxSetup({'async': true});
    $.post('ajax.php', {'operation': 'updateSite', 'statusId': statusId, 'param': paramValue}, function(data) {
        $('#progress_bar').stopTime('statusLog'); //long operation is finished - stop the timer
        if (data.result) {
            //operation probably successful
        } else {
            //operation failed
        }
    });
    //query for progress every 4s, 'statusLog' is just the name of the timer
    $('#progress_bar').everyTime('4s', 'statusLog', function() {
        var elm = $(this);
        $.post('ajax.php', {'operation': 'showLog', 'statusId': statusId}, function(data) {
            if (data) {
                //set bar percentage
                $('#progress').css('width', parseInt(data.progress) + '%');
            }
        });
    });
    return false;
}

Backend code (in PHP):

if (isset($_POST['operation'])) {
        ini_set("display_errors", false);
        session_write_close();  //otherwise requests would block each other
        switch ($_POST['operation']) {
            /**
            * Initialize progress operation, acquire ID (statusId) of that operation and pass it back to
            *   JS frontend. The frontend then sends the statusId back to get current state of progress of
            * a given operation.
            */
            case 'beginOperation': {
                $statusId = //insert into progress_data
                echo json_encode(array('statusId' => $statusId));
                break;
            }
            /**
            * Return back current progress state.
            */
            case 'showLog': {
                $result->progress = (float) //SELECT progress FROM progress_data WHERE statusId = $_POST['statusId']
                echo json_encode($result);
                break;
            }
            case 'updateSite': {
                //start long running operation, return whatever you want to, during the operation ocassionally do:
                    UPDATE progress_data SET progress=... WHERE statusId = $_POST['statusId']
            }
        }
    }
    /* Terminate script, since this 'view' has no template, there si nothing to display.
    */
    exit;

I have used this approach in 3 applications already and I must say it is very reliable and fast enogh (the showLog operation is just a simple SELECT statement). It is also possible to use session to store the progress, but that brings a lot of problems, since the session has to be write closed (if it is stored in files), otherwise the showLog AJAX queries will wait for the long operation to finish (and loose sense).

Answer 10

For showing progress during load, I would modify my backend so that it can do selective loading.

For example,

var total_rec = 5000;
var load_steps = 20;
var per_load = total_rev / load_steps;
var loaded = 0; 
while (loaded < total_rec) {
    www.foobar.com/api.php?start=loaded&end=(loaded+per_load);
    loaded += per_load;
}

Every time the load is done, update the progress bar.

An alternative way to modify the backend can be

www.foobar.com/api.php?start=loaded&count=50

Answer 11

Something seems fishy.

I'm not familiar with databases that can't update 5000 records in a flash... So this isn't just a single update you're applying but a record by record update?

Consider a system which lets a user download 5000 entries and does not mark which ones have been edited and then at the end of making updates applied a single update button which would require that all 5000 records be passed back in some way. That would be worst case.

So there may be some way to partition the problem such that there is no wait time. Consider for instance a temporary db table (or just in the applications memory, pretty easy by just creating a list of ORM entities... but that's an aside) then when it comes time to commit those updates they can at least be done in a batch without requiring any client/server transfer. It may even be possible to mark the individual fields which were edited so there is nothing more updated on the DB except exactly what changed.

There are long running processes and I know sometimes you're stuck having to use what someone has provided you with... but perhaps with a bit of thinking you can simply get rid of the wait time.

Answer 12

I once did something similar this way (similar to Zain Shaikh, but simpler):

On the server

int toUpdate = 5000;  
int updated = 0;  
int prev = updated;
while(updated < toUpdate)  
{  
    updated = getAlreadyUpdatedRows();
    flushToClient(generateZeroSequenceOfLength(updated - prev));  
    prev = updated;  
    sleep(500);  
}  
closeStream();

On the client
Follow the Zain Shaikh path but on ProcessInput simply resize your progress bar according to the ratio between the number of records to update and the input.length.

This solution often trades client side complexity for network bandwidth.

Do not mix this server activity with an eventual file import, unless you really know what you are doing. You would be trading db queries (the select for counting the updated rows) for stability: what if the user change page? There's no problem for the progress bar, but the import would not be completed.