I am trying to use Selenium to scrape a e-commerce website www.flipkart.com Here are the steps I am trying for:
1. Open Flipkart
2. Close the pop-up login prompt
3. Go to the Search Menu
4. Search for a keyword "Laptop"8/*/-
5. Click on the Search button
6. Open every record as a new tab and scrape information from it.
7. After scraping close the newly opened tab and go to the parent tab.
8. Repeat Steps for all the records on the page.
9. After getting all the records from one page go to the next page and repeat steps.
The problem I have with my code is that I am unable to close the newly opened tab from step 6.
driver=webdriver.Chrome()
driver.get("http://www.flipkart.com")
#Close pop us
elem=driver.find_element_by_xpath("/html/body/div[2]/div/div/button")
elem.click()
#click on search field
#1
elem=driver.find_element_by_xpath("xpath_tag_too_long_to_add")
elem.send_keys("Laptop")
#search
#2
elem=driver.find_element_by_xpath("xpath_tag_too_long_to_add")
elem.click()
time.sleep(10)
#get the maximum page number
y=int(driver.find_element_by_css_selector('div._2zg3yZ').text[10:12])
#scrape pages
name=[]
price=[]
color=[]
for i in range(1,y+1):
driver.get(link+str(i))
x=driver.find_elements_by_css_selector('div._3wU53n')
for i in x:
i.click()
actions = ActionChains(driver)
driver.switch_to.window(driver.window_handles[1])
time.sleep(2)
#3
names=driver.find_element_by_xpath('xpath_tag_too_long_to_add').text
name.append(name)
#4
prices=driver.find_element_by_xpath('xpath_tag_too_long_to_add').text
price.append(prices)
#5
colors=driver.find_element_by_xpath('xpath_tag_too_long_to_add').text
color.append(colors)
driver.switch_to.window(driver.window_handles[0])
I have tried things like action keys to close a tab, but it has not worked. I problem I get because of not closing the newly opened tab is that when I use switch_to.window it keeps going back and forth with the parent page and the first opened page giving me the same value in all the columns.
#this is what I have tried
elem=driver.find_element_by_tag_name("body")
elem.send_keys(Keys.CONTROL+"w")
Here are the Xpath & Links:
#1-/html/body/div[1]/div/div[1]/div[1]/div[2]/div[2]/form/div/div/input
#2-/html/body/div[1]/div/div[1]/div[1]/div[2]/div[2]/form/div/button
#3-/html/body/div[1]/div/div[3]/div[2]/div[1]/div[2]/div[2]/div/div[1]/h1/span
#4-/html/body/div[1]/div/div[3]/div[2]/div[1]/div[2]/div[2]/div/div[3]/div[1]/div/div[1]
#5-/html/body/div[1]/div/div[3]/div[2]/div[1]/div[2]/div[7]/div[3]
/div/div[2]/div[1]/div[1]/table/tbody/tr[5]/td[2]/ul/li
Link to Website: https://www.flipkart.com/
Thanks for reading.