How can I get a intermediate URL from a redirect chain from Selenium using Python?

Question

I'm using Selenium with Python API and Firefox to do some automatic stuff, and here's my problem:

1. Click a link on original page, let's say on page a.com
2. I'm redirected to b.com/some/path?arg=value
3. And immediately I'm redirected again to the final address c.com

So is there a way to get the intermediate redirect URL b.com/some/path?arg=value with Selenium Python API? I tried driver.current_url but when the browser is on b.com, seems the browser is still under loading and the result returned only if the final address c.com is loaded.

Another question is that is there a way to add some event handlers to Selenium for like URL-change? Phantomjs has the capacity but I'm not sure for Selenium.

Answer 1

You can get redirects from performance logs. According to docs and github answer here is what I've done in C#, should be possible to port in Python:

var options = new ChromeOptions();
var cap = DesiredCapabilities.Chrome();
var perfLogPrefs = new ChromePerformanceLoggingPreferences();
perfLogPrefs.AddTracingCategories(new string[] { "devtools.network" });
options.PerformanceLoggingPreferences = perfLogPrefs;
options.AddAdditionalCapability(CapabilityType.EnableProfiling, true, true);
options.SetLoggingPreference("performance", LogLevel.All);
var driver = new ChromeDriver(options);
var url = "https://some-website-that-will-redirect.com/";
driver.Navigate().GoToUrl(url);
var logs = driver.Manage().Logs.GetLog("performance"); //all your logs with redirects will be here

Looping through logs, if message.params.redirectResponse.url is equal to original URL then message.params.request.url will contain redirect URL

Answer 2

Proxy Servers such as BrowserMob proxy can be setup into your Selenium test and then have your web traffic routed via the the Proxy server. The traffic information is all captured as HAR files.You can try getting this information by plugging in a proxy server such as BrowserMob Proxy

AFAIK The only listening hook in mechanism that Selenium provides is the EventFiringWebDriver wherein you can plugin your own event listening by extending AbstractWebDriverEventListener via the register method in EventFiringWebDriver. But the EventFiringWebDriver has limitations. It cannot eavesdrop into events that arise out of Actions class. There's an alternative to that as well. Sometime back I created a blog post that talks about it. Maybe you can refer that as well. Here's the link

I don't know if there is similar to this in Python (since I have never worked with the Selenium Python bindings )

Answer 3

Answer my own question.

If the redirect chain is very long, consider to try the methods @alecxe and @Krishnan provided. But in this specific case, I've found a much easier workaround:

When the page finally landed c.com, use driver.execute_script('return window.document.referrer') to get the intermediate URL

Answer 4

is there a way to get the intermediate redirect URL b.com/some/path?arg=value with Selenium Python API?

I would use an Explicit Wait with a small poll interval. The idea would be to wait for the staleness of the body element on the initial page:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

body = driver.find_element_by_tag_name("body")

wait = WebDriverWait(driver, 5, poll_frequency=0.05)
wait.until(EC.staleness_of(body))
print(driver.current_url)

You might also need to decrease the page load timeout:

driver.set_page_load_timeout(0.5)

Another question is that is there a way to add some event handlers to Selenium for like URL-change?

This is exactly what these Explicit Waits are about. There are relevant title_is, title_contains expected conditions and it's easy to write your custom one (for example, to wait for some substring in the current URL).