Python: Tell BeautifulSoup 'IF' Value not found, retry

Question

I am still a beginner in python and I am trying to scrape a website, I would like Python to search for a value until it is visible, therefore retry until the value is available rather than giving error 'AttributeError: 'ResultSet' object has no attribute 'get' if no value is found. This is my code right now:

resumeURL='url'
response=self.session.get(resumeURL,headers=headers)
soup=BeautifulSoup(response.content, "html.parser")

form=soup.find_all('input',{'name':'form_id', 'type':'hidden'})
for form in form:
print(form.get('value'))

Short answer: Use selenium to load dynamic content Ref: https://stackoverflow.com/questions/17597424/how-to-retrieve-the-values-of-dynamic-html-content-using-python — northtree, Feb 27 '18 at 23:25

score 0 · Answer 1 · answered Feb 28 '18 at 04:52

0

You have to render dynamic content.

Options:

selenium, https://github.com/SeleniumHQ/selenium
Requests-HTML: HTML Parsing for Humans, https://github.com/kennethreitz/requests-html#requests-html-html-parsing-for-humans

answered Feb 28 '18 at 04:52

northtree

8,569
11
61
80

Python: Tell BeautifulSoup 'IF' Value not found, retry

1 Answers1