Dynamic progress bar based on progress of script?

Question

I have a Google App Engine application, using the webapp2 framework, that interacts with a MySQL database. Users of the application can upload data. During uploading, I want to show a progress bar, since it can take up to some minutes.
Based on what I've seen in other topics (mainly: this topic and this one), I'm working on a JSON/Javascript solution, which are both new to me.

The progress bar itself is working if I'm passing a random number. However, I can't figure out how to 'load' the changing values from the Python script.

Here's the HTML/CSS/Javascript:

HTML:
<div id="myProgress">
    <div id="myBar"</div>
</div>  

CSS:
#myProgress {width: 300px;
height: 30px;
background-color: #ddd;
}
#myBar {width: 1%;
height: 30px;
background-color: #4CAF50;
}   

Javascript:     
<script type="text/javascript">
function move() {
    var elem = document.getElementById("myBar");   
    var width = 1;
    var id = setInterval(frame, 1000);
    function frame() {
        if (width >= 100) {
            clearInterval(id);
        } 
        else {
            //var randomnumber = Math.floor(Math.random() * 100); --> works
            var randomnumber = function update_values() {
                $SCRIPT_ROOT = {{ script_root }};
                $.getJSON($SCRIPT_ROOT+"/uploading",
                    function(data) {
                        $("#width").text(data.width+" %")
                    });
            } ; --> this is wrong I assume
            var width = randomnumber; 
            elem.style.width = width + '%'; 
        }
    }
}
window.onload=move();   
</script>

The progress comes from a for loop in Python which is embedded in the script that loads the page. After the script is finished with one activity, I want the result of counter to be passed to the progress bar as its width. With static variables, I use the regular Jinja way.

class UploadingpageHandler(webapp2.RequestHandler):
def get(self):
    activities_list = [1,2,3,4,5,6,7,8,9,10]
    counter = 0
    script_root = 'localhost:10080'

    for activity in activities_list:
        counter = counter + 10
        upload.do_stuff_in_some_function_with_MySQL()   
        obj = {
        'width': counter
        }
        self.response.headers['Content-Type'] = 'application/json' --> this
        self.response.out.write(json.dumps(obj)) --> and this is wrong I assume

    template_vars = {
    'script_root': script_root
    }
    template = jinja_environment.get_template('uploading.html')
    self.response.out.write(template.render(template_vars))

How to alter the scripts to get it working? Or is there a better way to solve this?

A simple ajax upload with progress bar... https://stackoverflow.com/questions/26674575/php-upload-extract-and-progressbar/26679480#26679480 this example posts to PHP but with a few alterations I'm sure it will work all the same for you. — NewToJS, Jun 01 '17 at 23:32
Thanks for your reply. I'm still not sure how to alter the proposed solution though...Can you guide me a little bit more on which parts to alter to get the right variables in the javascript? And I'm also looking for a way to load the value from the Python script. — Pieter, Jun 01 '17 at 23:57
If you're uploading something you don't need to have python pass anything, just have it process the upload, I have added event listeners to listen for the process of the upload. So `runprogress(event)` will calculate the process of 0-100 so if you add console.log(percent) you will see it log 0 to 100. `uploadcomplete(event)` will execute once the upload is complete so you could use that to set a status message and clear/reset your progress bar. As for the changes to the ajax post you should be able to change `upload.php` to the server-side file you want to use. — NewToJS, Jun 02 '17 at 00:02
This should work all the same for `POST` and `GET` methods, this will calculate the the progress of the ajax call in place. — NewToJS, Jun 02 '17 at 00:03
Ah, I get it. One more question to make sure this will lead to a solution. After a user clicks on 'upload', a python request is made to the API of another website. So it's not the user himself who is uploading a file, like in the example, it's the progress of a serie of requests which I want to show in the progress bar. Does it still work then? — Pieter, Jun 02 '17 at 00:15
If the data is passed via Ajax then ajax will have to wait for a reply from the call.... this is calculating the response. For example, uploading a photo.... the client selected a photo, ajax posts that photo to a server-side file... ajax knows the server-side is processing it so the event I have in there is listening to that and updating when that process changes hence the 0-100. — NewToJS, Jun 02 '17 at 00:19

score 1 · Accepted Answer · answered Jun 01 '17 at 23:57

1

You need to store the progress of your "activities" outside of your function somewhere.

A hacky "solution" would be to store it into some sort of caching solution like memcached or redis with some sort of timestamp/signature so that you can retrieve it (and invalidate the old entries with a cron job-type thing).

Or you could go balls out and make your task entirely async with something like Celery, but I doubt you can do that on Google App Engine.

answered Jun 01 '17 at 23:57

airstrike

2,270
1
25
26

1

Thanks! I think i've tried Celery before. If I recall correctly, it doesn't work with GAE. GAE does have there own task queue thing (https://cloud.google.com/appengine/docs/standard/python/taskqueue/). Does that do the same? I will look into memcached and redis as well if not... – Pieter Jun 02 '17 at 00:18
That GAE Task Queue seems to be just what you need. Because this is a series of requests, I think the best solution really is the task queue but developing that is a lengthy exercise. I'd weigh the costs of developing against the real need for the progress bar. Is this a major component of your website? If so, handling those tasks (including the ability to retry etc) is key. If not, then maybe rethink what you're trying to do from a fresh approach or give up on the progress bar altogether. – airstrike Jun 02 '17 at 01:00
One such novel approach would be to break up the 10 tasks (or activities as you called them) into 10 separate entry points in your API and then handle the "scheduling" of those tasks on the client side rather than as you currently have, which is a loop in the Python function. Basically, once task 1 is finished (an event that you can trigger through a callback on your ajax request), start task 2. This way you trade the cost of developing a Python task queue system for complexity in the JS code (and you probably lose some functionality such as retrying tasks) – airstrike Jun 02 '17 at 01:06
Thanks! Rethinking if its really necessary is the key here, good to bring that up :) I think the solution will be to estimate the time it will cost, send that figure as a static number to the template and let the javascript increment the bar at a set interval. But not after I tried to get the Ajax solution mentioned by @NewToJS first – Pieter Jun 02 '17 at 14:31

Dynamic progress bar based on progress of script?

1 Answers1