Can't get XPath to click on pop up in selenium

Question

I'm trying to extract a few simple 'CIK' codes from the 'SEC'. If you run the code below, you will get a pop about a "survey". If you do it manually, you won't see it. It bombs my code. But since it is in selenium, I can't inspect it with chropath to get the xpath to click on the "NO". And I can't recreate the pop up in a normal browser. What do I do?

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from random import randint

path ='C:\\Users\\Jason\\Google Drive\\python\\chromedriver.exe' 
ticker='alrm'
main='https://www.sec.gov/search/search.htm'
driver=webdriver.Chrome(path)

tickers=['AAL','AAN','AAOI','AAPL']
# starts the process
def get_CIK(ticker):
    driver.get(main)
    stock_code = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "cik")))
    stock_code.click()
    stock_code.send_keys(ticker)
    
    
    driver.find_element_by_xpath("//input[@value='Find Companies']").click() # click on search buttom
    link = driver.find_element_by_xpath("/html[1]/body[1]/div[4]/div[1]/div[3]/span[1]/a[1]").get_attribute("href") # get link
    cik= link[link.index('CIK=')+4:link.index("owner")-1] # extract cik
    print cik
    
for i in tickers:
    get_CIK(i)

Answer 1

What you occasionally see is the "Foresee popup" which would generally show up randomly.

There are 5 generic approaches I can think of:

• set a specific cookie that would disable foresee popup "pretending" you've dismissed it already. Which cookie to set exactly is an open question at the moment. There is this relevant thread about it as well: Handle random ForeSee popup using Python and Selenium

• check for the presence of the "popup" all the way during the interaction with the web-site. The popup is not a classic "alert" in selenium sense and is just an "overlay" which has this HTML representation:

  <div class="__acs " aria-labelledby="fsrHeading">
      <div class="acsModalBackdrop acsAbandonButton" data-isbackdrop="true">
          <div class="acsFocusFirst acsClassicInvite" tabindex="1"
               id="acsMainInvite" role="dialog" aria-labelledby="fsrHeading">
              <div class="acsClassicInner" role="document">
                  <div class="acsLogoSpacer"><img
                      src="//gateway.foresee.com/sites/sec-gov/production/trigger/sitelogo.png"
                      class="acsSiteLogo" title="" alt=""> <img
                      src="https://static.foresee.com/logos/foresee/150_67.png"
                      class="acsNoDisplay" title="ForeSee" alt="ForeSee">
                      <div title="ForeSee" alt="ForeSee"
                          class="acsVendorLogoSVG"></div>
                          ... 
  

  You can then, for example, check for the presence of the "Decline" button and click it if it's present:

  <a href="#" tabindex="2" class="acsInviteButton acsDeclineButton" title="" role="button"></a>
  

• you can also "block" foresee JS scripts to be loaded by, for example, using a browsermobproxy where you would block all traffic from "foresee.com". Or, on a similar trajectory - you can start selenium with an ad blocker which would/should block "foresee" from out-of-the-box

• or, you can override the showInvite() function of the "foresee" global object:

  driver.execute_script("window.FSR.showInvite = function () {};")
  

  Note that you would need to call this script every time right after you navigate to a new page.

• after a bit of a reverse engineering I found that "foresee" JS object has this global config which contains lots of interesting information including a device blacklist:

  device_blacklist: ["HTC_Rezound", "blackberry"]
  

  You can then override the browser's user-agent and pretend to be coming from a, say, Blackberry device:

  BLACKBERRY_UA = "Mozilla/5.0 (BlackBerry; U; BlackBerry 9900; en) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.1.0.346 Mobile Safari/534.11+"
  
  opts = Options()
  opts.add_argument("user-agent={0}".format(BLACKBERRY_UA))
  
  driver = webdriver.Chrome(chrome_options=opts)
  

---

The second option is though technically more challenging and more error-prone, and would slow things down as you would constantly check for this popup to be present. Well, at least until you dismiss it.

The fourth option is quite promising but I have not fully tested it.

The last option, regardless of how crazy it sounds, actually works for me.

Answer 2

In a nutshell

When following script is executed in the browser's console -

driver.execute_script("window.FSR.setFSRVisibility(true);")

it makes ForeSee popup appear behind the rest of HTML elements. And doesn't affect UI tests anymore

---

Theory

So ForeSee is one of those services that can be integrated with any web app, and will be pulling js code from their API and changing HTML of your app, by executing the code on the scope of the website. Another example of such company is walkme

Obviously in modern world, if these guys can overlay a webpage, they should have a configuration to make it optional (at least for lower environments) and they actually do. What I mentioned as a solution came from this page. But assuming they didn't have such option, one could reach out their support and ask how to workaround their popups. Even if they didn't have such option they would gladly consider it as a feature for improvement.