Web scraper to collect data into Excel

Question

I'm stuck on my first ever web scarper, it opens the browser, but does nothing then, could you help me out please?

from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd

driver = webdriver.Chrome('C:\Program Files\Google\Chrome\Application\chrome.exe')
products=[]
prices=[]
ratings=[]
driver.get = ("<a href=https://www.flipkart.com/laptops-store?otracker=nmenu_sub_Electronics_0_Laptops")

content = driver.page_source
soup = BeautifulSoup(content)
for a in soup.findAll('a', href=True, attrs={'class' : '_3YgSsQ'}):
    name=a.find('div', attrs={'class' : '_2rpwqI'})
    price = a.find('div', attrs={'class': '_30jeq3'})
    rating = a.find('div', attrs={'class': '_3LWZlK _1wB99o'})
    products.append(name.text)
    prices.append(price.text)
    ratings.append(rating.text)

    df = pd.DataFrame({'Product Name': products, 'Price': prices, 'Rating': ratings})
    df.to_csv('products.csv', index=False, encoding='utf-8')```

Shouldn't you be calling `driver.get` with a URL, rather than assigning the string representation of an element to it? Compare what you've posted to e.g. https://selenium-python.readthedocs.io/getting-started.html#simple-usage. — jonrsharpe, Feb 04 '22 at 13:28

undetected Selenium · Answer 1 · 2022-02-04T15:07:38.233

get(url: str)

get(url) loads a web page in the current browser session, which accepts a url: string

As per the line of code:

driver.get = ("<a href=https://www.flipkart.com/laptops-store?otracker=nmenu_sub_Electronics_0_Laptops")

you are passing the partial HTML markup of the desired WebElement which would result in an error. Instead you need to pass the url: str as follows:

driver.get = ("https://www.flipkart.com/laptops-store?otracker=nmenu_sub_Electronics_0_Laptops")

Web scraper to collect data into Excel

1 Answers1

get(url: str)