2

I am struggling with an application I want to make, which is a web application that will scrape an other website using the selenium package with chromedriver. I am using python 3.7 on elementary OS. Everything works okay if I don't include the loading page, even if I didn't put it online yet so I dunno for sure, but my problem is that I would like to let the crawling headless and no sandbox, while having the waiting page in my app. I'd like to render to a success html template when the job is done, too.

I searched on stackoverflow for answers, but I couldn't make my thing work. (sorry if duplicate) One thing that I found is Luiz Aoqui's answer to this question : Flask is not render_template before executing long function , and it seems like the OP could solve his issue. I couldn't, though. I don't know javascript at all so maybe you will find this very simple, if the problem came from here.

The python flask code : 

@app.route('/auto_connect/', methods=["GET", "POST"])
def connect():
    if session['mail'] != None:
        if request.method == "POST":
            session['job'] = request.form['job']

            return redirect(url_for('process', fun='auto_connect'))
        return render_template('auto_connect.html')
    return redirect(url_for('login'))   

@app.route("/process/<fun>", methods=['GET', 'POST'])
def process(fun, *args):

    if fun == 'auto_connect' or fun == 'auto_apply':
        if request.method == 'GET':
            return render_template('wait.html', fun=fun)

        if request.method == 'POST':
            print('test')
            if fun == 'auto_connect':
                auto_connector(session['mail'], session['password'], session['job'])
                return 'done'

            elif fun == 'auto_apply':
                auto_applyer(session['mail'], session['password'], session['job'], session['location'])
                return 'done'
            else:
                return "error"
    return 'error'

The JS code : 

var request = new XMLHttpRequest();
request.open('POST', '/process/'.concat({{fun}}));

request.onload = function() {
  if (request.status === 200 && request.responseText === 'done') {
    // long process finished successfully, redirect user
    window.location = '/success/' ;
  } else {
    // ops, we got an error from the server
    alert('Something went wrong. FROM server');
  }
};

request.onerror = function() {
  // ops, we got an error trying to talk to the server
  alert('Something went wrong. TO server');
};

request.send();

The loading page is showing, but the scraping does not start. I expect it to start with the "open" POST request in the js code which is part of the 'wait.html' template.

PS : there is the flask debugger, I put a print at the top of my scraping script which shows up in terminal when I do no render the loading page, but doesn't when I do.

127.0.0.1 - - [04/Aug/2019 02:03:20] "GET /auto_connect/ HTTP/1.1" 200 -
127.0.0.1 - - [04/Aug/2019 02:03:22] "POST /auto_connect/ HTTP/1.1" 302 -
127.0.0.1 - - [04/Aug/2019 02:03:22] "GET /process/auto_connect HTTP/1.1" 200 -

Thanks in advance for your answers.

DMarwan
  • 21
  • 2

1 Answers1

0

I found the answer myself to my question. I am not a web programmer so I lack some automatism sometimes, but by inspecting my waiting page, I saw an error in the JS script in the concat function. Calling '{{fun}}' in the script instead of {{fun}} did the thing.

I didn't understand this behavior since my object fun is a python string, but I read the jinja2 doc and all became lucid to me, we can see in this link https://jinja.palletsprojects.com/en/2.10.x/templates/ the following:

{{ ... }} for Expressions to print to the template output

This prints the value, so it is no longer a string but raw text in html template. (type(print(x)) is None in python3 btw)

If this helps someone then I'm glad.

DMarwan
  • 21
  • 2