User input from HTML for use with Python and Selenium

Question

I have a python application that uses PySimpleGUI to take user input (name, date, and file), navigate a website, and enter data based on the given input. I'm looking to host this program on pythonanywhere, but since the gui isn't supported, I need to find another way to get the user's input. I've created an HTML web form, but am unsure how to proceed.

1) Will the use of request.get work to obtain the name, date, and file?

2) Selenium doesn't control the web page opened by submitting the HTML form. I need to explicitly call browser(open). Since Selenium isn't tied to the HTML page that opens, the code is executed before the web form is submitted and I get a timeout exception. With the GUI previously used, the remaining code wouldn't execute until after submission (which is ideal).

I want to either:

use Selenium to control the web page opened by submitting the HTML form

OR

stop my code from running before the HTML form has been submitted. Once submitted, close the form and run the code.

Any thoughts?

Below I've included the HTML file, the Python code I'm trying to get to work, and the relevant portion of the program using PySimpleGUI that functions as desired.

<!DOCTYPE html>
<html>
<body>

<label for="opt_select">Select an Option:</label>
<select id="opt"
        name="opt_list">
  <option value="AA">AA</option>
  <option value="BB">BB</option>
  <option value="CC">CC</option>
</select>

</body>

<form action="/action_page.php">
  <label>Date:</label>
  <input type="date" id="sum_date" name="sum_start">
</form>
</body>
<form target="_blank" action="http://awebsite"
      method="post" id="submitted_form"
      name="submission" class="validate">
</form>
<form>
  <label for="myfile">Select a file:</label>
  <input type="file" id="myfile" name="myfile"><br><br>
  <input type="submit">
</form>
</html>

    import pandas as pd
    from datetime import datetime
    from selenium import webdriver
    from selenium.webdriver.common.keys import Keys
    from selenium.webdriver.common.action_chains import ActionChains
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.common.by import By
    from flask import Flask, render_template, request

    app = Flask(__name__)

    @app.route('/hello', methods=['POST'])
    def hello():
        return render_template('gui.html')

    def user_data():
        data = request.GET['opt_list']

    browser = webdriver.Chrome()
    browser.get('https:/awebsite')
    type(browser)
    delay = 20

    if data == "AA":
        WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'AALanding')))
        browser.find_element_by_id('AALanding').click()
    if data == "BB":
        WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'BBLanding')))
        browser.find_element_by_id('BBLanding').click()
    if data == "CC":
        WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'CCLanding')))
        browser.find_element_by_id('CCLanding').click()

    if __name__ == '__main__':
        app.run(host='0.0.0.0', port=3000)

import tkinter as tk
import time
import pandas as pd
import PySimpleGUI as sg
from datetime import datetime
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

sg.theme('DARKTEAL')
layout = [[sg.Text('Select Student', key='-STUOUTPUT-', font=('Arial', 10))],
          [sg.Combo(['AA', 'BB', 'CC', 'DD'], size=(5, None), font=('Arial', 10),
                    key='-OPT-')],
          [sg.Text('Date to Start Summing', font=('Arial', 10), visible=True),
           sg.In(key='-CAL-', size=(12, None), default_text='MM/DD/YYYY')],
          [sg.CalendarButton('Calendar', key='-OUTPUT-', target='-CAL-', pad=None, size=(10, None), font=('Arial', 10),
                             format='%m/%d/%Y')],
          [sg.Text('Filename', key='-FOUTPUT-', font=('Arial', 10))],
          [sg.In(visible=False),
           sg.Input(key='-DIR-', size=(20, None)),
           sg.FileBrowse('Browse', target='-DIR-', font=('Arial', 10))],
          [sg.OK(font=('Arial', 10)), sg.Cancel(font=('Arial', 10))]]

window = sg.Window('Data Collector', layout, grab_anywhere=False, size=(400, 280), return_keyboard_events=True,
                   finalize=True)

event, values = window.read()
window['-STUOUTPUT-'](values['-STUIN-'])
window['-OUTPUT-'](values['-CAL-'])
window['-FOUTPUT-'](values['-DIR-'])
acedate = (values['-CAL-'])
opt = (values['-OPT-'])
file = (values['-DIR-'])
window.close()

browser = webdriver.Chrome()
browser.get('awebsite')
type(browser)

delay = 20

df = pd.read_excel(file, Sheet_name=0, header=None)

def bx_select():
    if data == "AA":
    WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'AALanding')))
    browser.find_element_by_id('AALanding').click()
if data == "BB":
    WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'BBLanding')))
    browser.find_element_by_id('BBLanding').click()
if data == "CC":
    WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'CCLanding')))
    browser.find_element_by_id('CCLanding').click()

def clear():
    ActionChains(browser) \
        .send_keys(Keys.CLEAR) \
        .perform()
def tab():
    ActionChains(browser) \
        .send_keys(Keys.TAB) \
        .perform()
def enter():
    ActionChains(browser) \
        .send_keys(Keys.ENTER) \
        .perform()
def page_up():
    ActionChains(browser) \
        .send_keys(Keys.CONTROL + Keys.HOME) \
        .perform()

bx_select()

clear()


def autosum():
    # length of bx measures
    x = len(df.columns)
    # used to determine when at end of row
    z = 1
    # location of column to start summing
    b = 1
    # number of days in the month
    c = len(df.index)
    # used to stop once last day of month reached
    y = 1
    # slices date chosen from calendar (dd) to determine where to start summing
    n = int(values['-CAL-'][3:5])
    # today's date (used to stop once current day is reached)
    d = datetime.today().strftime('%Y-%m-%d')

    while n < c:
        while z < x:
            m = df.iloc[n, b]
            z = z + 1
            b = b + 1
            if pd.isnull(m):
                tab()
                continue
            else:
                ActionChains(browser) \
                    .send_keys(str(m)) \
                    .perform()
            if z == x:
                n = n + 1
                y = y + 1
                z = 1
                b = 1
                enter()
                page_up()
                if n == c or str(pd.to_datetime(df.iloc[n, 0]).date()) == d:
                    return
                WebDriverWait(browser, delay).until(EC.visibility_of_element_located(
                    (By.CSS_SELECTOR,
                     'div.ZForm:nth-child(5) > div:nth-child(1) > div:nth-child(1) > h5:nth-child(1)')))
                bx_select()
                clear()
            else:
                tab()
                time.sleep(0.5)
autosum()

The code at the top is the HTML that lives in a file called gui.html — Alex Elfont, May 03 '20 at 20:47
Are you sure chrome browser will run on the python anywhere server? — Tarun Lalwani, May 04 '20 at 08:27
Looks like pythonanywhere only supports Firefox, but heroku supports Chrome — Alex Elfont, May 04 '20 at 10:52
PythonAnywhere dev here -- we have a beta running where you can get Chrome support added to your account. Just drop us a line using the "Send feedback" link while logged in to our site, and we can switch it on for you. — Giles Thomas, May 05 '20 at 10:14
@ Giles Thomas thanks for the heads up. Do you happen to know if the GUI shown above or a GUI using tkinter would work if using `pyvirtualdisplay`? — Alex Elfont, May 05 '20 at 14:42

score 0 · Answer 1 · answered May 09 '20 at 23:39

So it seems to me that this question is more about Flask than selenium or HTML. You seem to want to allow the user to input data on a HTML form and for that form to call some function in your python script. You can do this not too difficultly. I believe that this question addresses your problem well.

Once you have your form call your desired script, you can use the input data on that method to start selenium and have it enter your data in the desired place.

Is this what you are looking for?

BinaryMonkey · Answer 2 · 2020-05-11T16:57:46.927

To answer 1)

You need to fix the HTML file and put all the input fields (those you need in process_form() below) in the same <form action="..."> tag, pointing to your Flask submission handler (see below). For instance, if you want to process 'opt_list', you need to move it under a <form> tag. I also recommend using the POST method (instead of GET). So the structure of the gui.html should look something like this:

<!DOCTYPE html>
<html>
<body>
<form action="/process_url" method="post" enctype="multipart/form-data">
  <label for="opt_select">Select an Option:</label>
  <select id="opt"
        name="opt_list">
    <option value="AA">AA</option>
    <option value="BB">BB</option>
    <option value="CC">CC</option>
  </select>
  <br/>
  <label>Date:</label>
  <input type="date" id="sum_date" name="sum_start">
  <br/>
  <label for="myfile">Select a file:</label>
  <input type="file" id="myfile" name="myfile"><br><br>
  <input type="submit">
</form>
</html>

To answer 2)

You need to move your Selenium code in a function that is called only when your form is submitted (not in the un-indented body of your script).

from flask import Flask, render_template, request

app = Flask(__name__)
app.config['MAX_CONTENT_LENGTH'] = 1 * 1024 * 1024


# This is the Flask handler to show your form
@app.route('/', methods=['GET'])
def hello():
    return render_template('gui.html')

# Add a Flask handler to process your form submission.
# (You can also use POST method instead of GET if you don't
# want your form content to show in the address bar)
@app.route('/process_url', methods=['POST'])  
def process_form():
    # Read form values
    opt = request.form['opt_list']
    sum_start = request.form['sum_start']
    myfile = request.files['myfile']
    file_content = myfile.stream.read().decode('utf-8', 'ignore')
    # Do Work
    my_work(opt, sum_start, file_content)
    # Return an HTML output of your results
    html = f"""
    <html>
      <body>
        <h3>Result page</h3>
        opt: {opt}<br/>
        sum_start: {sum_start}<br/>
        myfile.filename: {myfile.filename}<br/>
        myfile.stream: <br/>
        <pre>{file_content}</pre>
      </body>
    </html>
    """
    return html

def my_work(opt, sum_start, file_content):
  # TODO: Put your Selenium code here 
  pass

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=3000, debug=True)

Also, try to avoid long execution times of your Selenium code to prevent browser timeout.

With those guidelines, I hope you will be able to solve your problem.

@Alex, in case something was not clear, you can test a working copy of the Flask part here: https://repl.it/@menavid/FlaskElfont — BinaryMonkey, May 11 '20 at 16:52

User input from HTML for use with Python and Selenium

2 Answers2