Selenium: Scrape Data From Newly Opened Tab

Question

I'm scraping data from a page that has 20 links on it, each link opens an identically formatted new tab. Selenium clicks on the first link, which opens in a new tab, but then attempts to scrape from the initial tab.

Relevant code:

for y in range(1):
    table = browser.find_element_by_xpath("//table[@summary='Sections']//tbody")
    rows = table.find_elements_by_tag_name('tr') 
    for row in rows:
        if row.find_elements_by_tag_name('td'):
            linktext = row.find_elements_by_tag_name('td')[3].text
            print(linktext)
            browser.find_element_by_link_text(linktext).click()
            time.sleep(5)
            title = browser.find_element_by_xpath("//p[@id='VAR1']").text
            section = browser.find_element_by_xpath("//p[@id='VAR2']").text
            description = browser.find_element_by_xpath("//p[@id='VAR3']").text

This code fails on the title = browser... line because it can't find the xpath. I confirmed that the xpath ("//p[@id='VAR1']") exists on the newly opened tab. If I instead use an xpath that exists only on the main page (not the newly opened tab), then that line succeeds and the error moves to the sections =... line.

How do I redirect Selenium to scrape the newly opened tab? And once I've done that, I'm assuming I'll need to redirect it back to its initial tab to click the next link, or at a minimum close the newly opened tab.

[this](https://gist.github.com/lrhache/7686903) may be relevant to you. It describes a process for shifting focus to the new tab, then shifting back. — HelpfulHound, Sep 29 '19 at 16:48

CEH · Accepted Answer · 2019-09-29T19:16:20.320

1

You’ll need to switch window frames on driver.

driver.switch_to.window.(driver.window_handles[-1]);

Switching to the last window handle will switch to the most recently opened window.

edited Sep 29 '19 at 19:16

answered Sep 29 '19 at 17:06

CEH

5,701
2
16
40

1

perhaps you meant `driver.switch_to.window(driver.window_handles[-1])` – Corey Goldberg Sep 29 '19 at 17:16
It depends on the language. Some have built in list operator methods such as `Last()`. But `[-1]` is functionally the same. – CEH Sep 29 '19 at 17:49
the question was tagged Python – Corey Goldberg Sep 29 '19 at 18:37

Selenium: Scrape Data From Newly Opened Tab

1 Answers1