Questions tagged [mechanicalsoup]

MechanicalSoup is a Python library for automated web browsing and form handling. It is built on top of the requests and BeautifulSoup libraries.

MechanicalSoup is a Python library for automated web browsing and form handling. It is built on top of the requests and BeautifulSoup libraries.

Install with: pip install mechanicalsoup

Example usage:

import mechanicalsoup

# Connect to duckduckgo
browser = mechanicalsoup.StatefulBrowser()
browser.open("https://duckduckgo.com/")

# Fill-in the search form
browser.select_form('#search_form_homepage')
browser["q"] = "MechanicalSoup"
browser.submit_selected()

# Display the results
for l in browser.get_current_page().select('a.result__a'):
print(l.text, '->', l.attrs['href'])
112 questions
7
votes
2 answers

Login to a website via Python - how to deal with CSRF?

I'm using Python 3 for a script that will monitor updates in a user's profile on a webpage. The login to this site is protected by CSRF countermeasures, which is a good thing. However, I can't get my script to login to this site. My approach using…
Scolytus
  • 16,338
  • 6
  • 46
  • 69
6
votes
2 answers

Using mechanicalsoup to set value of form element w/o a name

I have searched through all mechanicalsoup & beautifulsoup documentation but can't figure out how to set the value of a form element using 'id' (because it doesn't have a name). import mechanicalsoup browser =…
WillyMack33
  • 101
  • 1
  • 7
5
votes
0 answers

Python mechanicalsoup Set Combobox Value

There is a dropdownlist in html page. I need to set value as I want using mechanicalsoup in Python. I tried this but it did not work. If it's possible(I hope) how can i set a value in combobox. There is also another combobox (named combo2). the…
Cumhur Ata
  • 809
  • 6
  • 18
3
votes
0 answers

Web Scrape Vanguard Investment Account (login)

I am trying to automatically scrape my balance info from my Vanguard investment account. I am stuck trying to login to the account. I've tried requests and mechanicalsoup and so far I have not been able to figure this out. import…
John
  • 43
  • 5
3
votes
1 answer

How to add a cookie to existing cookies in MechanicalSoup

I know MechanicalSoup has a function called set_cookiejar() but it replaces the current cookiejar completely. I want to know how to add new cookies to existing cookies.
3
votes
2 answers

Download file with mechanicalsoup

I want to download the Excel file on this ONS webpage using the MechanicalSoup package in Python. I have read the MechanicalSoup documentation. I have searched extensively for an example to follow, on StackOverflow and elsewhere, without luck. My…
Ian
  • 33
  • 4
3
votes
1 answer

Data Scraping from pogdesign.co.uk/cat/

I am trying to scrape some data from http://www.pogdesign.co.uk/cat/. I want to get the channel and the air-time of each program, but the problem is that by default they do not appear. Only after manually configuring the settings and saving them,…
2
votes
2 answers

certificate verify failed with MechanicalSoup

I want to retrieve some data from my router using MechanicalSoup. However when I try I get "certificate verify failed". As I see it I have two options Find a way to disable SSL verification Tell MechanicalSoup where it can find the pem file (I…
2
votes
1 answer

How to follow a link that is actually a click button

I need to fill a form aproximately 100 times a day. I tried with selenium but it is quite slow because it is using a web browser. It's working quite well with mechanicalsoup until I need to click "Accept" button. Clicking it activates other tables…
Cmb
  • 31
  • 4
2
votes
0 answers

Filling