How to fill a form with post request and get response

Question

I have a form like following:

url = "http:/foo.com"
<table>
    <form action="showtree.jsp" method="post" id="form" name="form">
        <input type="hidden" id="sortAction" name="sortAction" value="">
        <tr>
            <td style="text-align: right;font-weight:bold;">State:&thinsp;</td>
            <td><select name="state">
                <option value="ca">CA</option>
                <option value="or">OR</option>
                <option value="al">AL</option>

                </select></td>
        </tr>
        <tr>
            <td style="text-align: right;font-weight:bold;">Populartion:&thinsp;</td>
            <td><select id="pop" name="population" onchange="disableShowOnAll()">
                <option value="100">100</option>
                <option value="200">200</option>
                <option value="300">300</option>


                </select></td>
        </tr>
        <tr>
            <td></td>
            <td>
                <button id="showbutton" class="btn btn-default" onclick="submitForm('show')">Show Tree
                </button>
            </td>
        </tr>
    </form>

So, basically the form has two options, State and Population and each has some options.. The idea is to select the options from the form and then submit.

On submit, the results are displayed in the same page..

So, basically how do i submit this post request in python...and then get the results (when the submit is pressed.. and the page is refreshed with the results?) Let me know if this makes sense? Thanks

Answer 1

What you're trying to do is submit a POST request to http://example.com/showtree.jsp

Using the requests library (recommended)

Reference: http://docs.python-requests.org/en/master/

The requests library greatly simplifies making HTTP requests, but is an extra dependency

import requests

# Create a dictionary containing the form elements and their values
data = {"state": "ca", "population": 100}

# POST to the remote endpoint. The Requests library will encode the
# data automatically
r = requests.post("http://example.com/showtree.js", data=data)

# Get the raw body text back
body_data = r.text

Using the inbuilt urllib

Relevant answer here: Python - make a POST request using Python 3 urllib

from urllib import request, parse

# Create a dictionary containing the form elements and their values
data = {"state": "ca", "population": 100}

# This encodes the data to application/x-www-form-urlencoded format
# then converts it to bytes, needed before using with request.Request
encoded_data = parse.urlencode(data).encode()

# POST to the remote endpoint
req =  request.Request("http://example.com/showtree.js", data=encoded_data)

# This will contain the response page
with request.urlopen(req) as resp:
    # Reads and decodes the body response data
    # Note: You will need to specify the correct response encoding
    #       if it is not utf-8
    body_data = resp.read().decode('utf-8')

Edit: Addendum

Added based on t.m.adam's comment, below

The above examples are a simplified way of submitting a POST request to most URI endpoints, such as APIs, or basic web pages.

However, there are a few common complications:

1) There are CSRF tokens

... or other hidden fields

Hidden fields will still be shown in the source code of a <form> (e.g. <input type="hidden" name="foo" value="bar">

If the hidden field stays the same value on every form load, then just include it in your standard data dictionary, i.e.

data = {
    ...
    "foo": "bar",
    ...
}

If the hidden field changes between page loads, e.g. a CSRF token, you must load the form's page first (e.g with a GET request), parse the response to get the value of the form element, then include it in your data dictionary

2) The page needs you to be logged in

...or some other circumstance that requires cookies.

Your best approach is to make a series of requests, to go through the steps needed before you would normally use the target page (e.g. submitting a POST request to a login form)

You will require the use of a "cookie jar". At this point I really start recommending the requests library; you can read more about cookie handling here

3) Javascript needs to be run on the target form

Occasionally forms require Javascript to be run before submitting them.

If you're unlucky enough to have such a form, unfortunately I recommend that you no longer use python, and switch to some kind of headless browser, like PhantomJS

(It is possible to control PhantomJS from Python, using a library like Selenium; but for simple projects it is likely easier to work directly with PhantomJS)