How to webscrape a popup on a new page with selenium?

Question

I would like to know how to webscrape a popup on a new page with selenium (or another framework). The Python code below clicks on a button then a new page opens but I don't know to copy/paste the promo code. Can you help ?

from selenium import webdriver
from time import sleep
from selenium.webdriver import ChromeOptions, Chrome




class Coupons_offers:



def stayOpen(self):
    opts = ChromeOptions()
    opts.add_experimental_option("detach", True)
    driver = Chrome(chrome_options=opts)

def __init__(self):
    self.driver = webdriver.Chrome()
    self.driver.get("https://www.offers.com/scotch-porter/")
    sleep(2)
    self.driver.find_element_by_xpath('//button[@id="_evidon-banner-acceptbutton"]').click()
    sleep(2)
    self.driver.find_element_by_xpath('//body/div[@id="main"]/div[1]/div[1]/div[3]/div[1]/div[1]/div[1]/div[3]/div[1]/div[1]/a[1]').click()
    sleep(4)
    self.driver.get("https://www.offers.com/scotch-porter/#offer_id=4467763")

score 2 · Answer 1 · answered Apr 27 '21 at 21:04

2

The modal window is loaded from external URL. You can use this code how to parse the coupon code (just replace the offerid for ID what you want):

import requests
from bs4 import BeautifulSoup

url = "https://www.offers.com/exit/modal/offerid/4467763/"  # <-- replace offerid here

soup = BeautifulSoup(requests.get(url).content, "html.parser")
print(soup.find(class_="coupon-code").text)

Prints:

SIGNUP15

answered Apr 27 '21 at 21:04

Andrej Kesely

168,389
15
48
91

Very nice way, I have to learn it ;) But he asked about Selenium approach – Prophet Apr 27 '21 at 21:08
@Prophet Well, he can load this URL with selenium....but.... ;) – Andrej Kesely Apr 27 '21 at 21:09
Again, I see how nice this way. I indeed want to learn this in the nearest future. But the way he is doing (and possible even doesn't know about this approach) is a legacy Selenium. – Prophet Apr 27 '21 at 21:11
1

Thank you for your help. it is easier with BS :) – Yo'python Apr 28 '21 at 13:49

score 0 · Accepted Answer · answered Apr 27 '21 at 21:02

By clicking self.driver.find_element_by_xpath('//body/div[@id="main"]/div[1]/div[1]/div[3]/div[1]/div[1]/div[1]/div[3]/div[1]/div[1]/a[1]').click() a new tab is opened.
Now you have to switch driver to the new tab with driver.switch_to.window(driver.window_handles[-1])
When on the new tab you can get the text from element located by //div[@class='coupon-code'] xpath
Now you can close the new opened tab by driver.close
And switch back to the previous tab by driver.switch_to.window(driver.window_handles[0]) See more details here

score 0 · Answer 3 · answered Apr 27 '21 at 21:03

It sounds like you will probably need to get the window handle of the new window and go to it. To debug, try

Whandles = driver.window_handles
print(Whandles)

And it should be clear what the window handle is for your new window. Then you can do

driver.switch_to_window_(desiredWindowHandle)

before proceeding with code to use the new window.

How to webscrape a popup on a new page with selenium?

3 Answers3