Save selected portion of website form output to csv with python

Question

I want to enter my origin and destination(O and D) into a form on this website (http://fahrplan.sbb.ch/bin/query.exe/en), then write the result to csv file. Automating this task is the only option I have since the number of locations to enter are close to 1000. Using the code below which I modified from here, I am able to input the entry locations in the form and print the result to my screen with br.response().read(). However, the result is printed in html format but I want the portion highlighted in blue in the image below exported to a csv file. How can I do that?

Image:

My Code:

from mechanize import Browser
br = Browser()

# Ignore robots.txt
br.set_handle_robots( False )
# Google demands a user-agent that isn't a robot
br.addheaders = [('User-agent', 'Chrome')]

# Retrieve the Google home page, saving the response
br.open('http://fahrplan.sbb.ch/bin/query.exe/en')

# # Show the available forms
# counter = 0
# for f in br.forms():
#     counter += 1
#     print f, counter
# print 'counter', counter

# Enter the text inpur
br.select_form(nr=6)
br.form[ "REQ0JourneyStopsS0G" ] = 'Leverkusen Mitte'
br.form[ "REQ0JourneyStopsZ0G" ] = 'Pescara Centrale'


# Get the search results
br.submit()
print br.response().read()


# How can I export the result to csv???

score 0 · Answer 1 · edited Jun 20 '20 at 09:12

If you look at the source of the resulting HTML page in Google's Chrome source console, you will find the four results. Here is a capture of the departure section of the first result:

You can get to the remaining results by searching the console using the text highlighted in yellow in my capture. All you need now is to scrape and slice this HTML code using Beautiful Soup, then save the sliced sections to a CSV file.

score 0 · Answer 2 · answered Jan 17 '17 at 11:53

As mentioned by another answer, you can parse the response using HTML parser such as BeautifulSoup, select each values that you want, put them in a comma-separated string then write it to file.

Here is the sample code to give you better idea:

from mechanize import Browser
from bs4 import BeautifulSoup

# get the response from mechanize Browser

soup = BeautifulSoup(response, 'html.parser')
trs = soup.select('table.hfs_overview tr')
with open('out.csv', 'a+') as f:
    for tr in trs:
        locations = tr.select('td.location.departure a')
        if len(locations) > 0:
            location = locations[0].contents[0].strip()
            prefix = tr.select('td.prefix')[0].contents[0].strip()
            time = tr.select('td.time')[0].contents[0].strip()
            # parse more values here
            # write to file
            f.write("{},{},{}\n".format(location, prefix, time))

Save selected portion of website form output to csv with python

2 Answers2