0

I am trying to pull some data from a website. Once I checked the data that I pulled with beuatifulsoup (using print(soup) in the code below) does not seem very well. It is different than once I check with view-source:URL. I am unable to find the fields that I am looking for. Could you please help me to find a solution?

Website: https://www.wayfair.com/furniture/pdp/mercury-row-stalvey-contemporary-4725-wide-1-drawer-server-w003245064.html

Basically, I am trying to get price of this product. I used the same code structure on other websites, it worked properly but it is not working on wayfair.

The second thing that I could not find a solution yet is the last line of my code (StyledBox-owpd5f-0 PriceV2__StyledPrice-sc-7ia31j-0 lkFBUo pl-Price-V2 pl-Price-V2--5000). Instead of name of the product is there a way to get only price like $389.99?

Thanks in advance!

This my code:

html = requests.get('https://www.wayfair.com/furniture/pdp/mercury-row-stalvey-contemporary-4725-wide-1-drawer-server-w003245064.html')
soup=BeautifulSoup(html.text,"html.parser")
print(soup)
inps=soup.find("div",class_="SFPrice").find_all("input")
for inp in inps:
    print(inp.get("StyledBox-owpd5f-0 PriceV2__StyledPrice-sc-7ia31j-0 lkFBUo pl-Price-V2 pl-Price-V2--5000"))
marista
  • 25
  • 5
  • Have you actually run your code? It should produce an error (as in my tests), because even the first "div" isn't in the requested html data. It's a client-rendered site! – e.d.n.a Jul 22 '21 at 14:12

2 Answers2

0

Try with:

soup.findAll('div', {'class': 'SFPrice'})[0].getText()

Or in a more simple way:

inps=soup.findAll('div', {'class': 'SFPrice'})[0]
inps.getText()

Both return the price for that specific product.

ILTRENTA
  • 61
  • 3
  • Thank you four the quick response. But both of them gives me the error below. – marista Jul 22 '21 at 12:00
  • inps=soup.findAll('div', {'class': 'SFPrice'})[0] IndexError: list index out of range – marista Jul 22 '21 at 12:00
  • That means there are none. That also explains why your original code didn't return anything. It was actually more correct. – tripleee Jul 22 '21 at 12:07
  • I did my example using Selenium. With requests i get the same error, and it is due to it being a client side rendered page as the answer below is stating. – ILTRENTA Jul 22 '21 at 14:45
0

Your site example is a client-side rendered page and the original html-data fetched doesn't include the searched for elements (like the div with class "SFPrice").

Check out this question for learning about how to scrape javascript-rendered pages with beautifulsoup in combination with selenium and phantomJS, dryscrape or other options.

Or you could also look at this guide.

e.d.n.a
  • 191
  • 1
  • 6
  • Please post a comment indicating the duplicate rather than posting a non-answer with a link to a different question. – tripleee Jul 22 '21 at 12:08
  • @tripleee How is it a duplicate, when he wasn't aware of the issue? So this answers his question, why he cannot find the element. – e.d.n.a Jul 22 '21 at 14:09
  • 1
    We can't require those who ask to know the answer. The guidance is that questions with the same *answer* should be closed as duplicates. – tripleee Jul 22 '21 at 16:14