6

So I have the following Selenium Chrome Webdriver process that is working correctly locally. Basically, when the below code is run, a purchase is made through Best Buy, and an SMS message is sent to me from my Twilio phone number indicating whether or not the purchase was successful. The variables in all-caps are already defined. I also did not include the imports. Again, the below code is making the purchase SUCCESSFULLY when run locally. This means that the SMS message that I get from my Twilio phone number contains "Success!" every time the following code is run locally:

client = Client(ACCOUNT_SID, AUTH_TOKEN)
def runBestBuyBotLocal():
    driver = webdriver.Chrome()
    driver.get("https://www.bestbuy.com/site/spongebob-squarepants-mini-plush-styles-may-vary/6404213.p?skuId=6404213")
    wait = WebDriverWait(driver, 10)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html/body[@class='size-l']/div[@class='pl-page-content']//div[@class='container-v2']/div[@class='row v-m-bottom-g']/div[2]//div[@class='col-xs-12']/div[6]/div[@class='None']/div/div/div/button[@type='button']")))
    element.send_keys(Keys.RETURN)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@class='c-portal']/div[@role='dialog']/div[1]/div[@role='dialog']/div[@role='document']//a[@role='button']")))
    element.send_keys(Keys.RETURN)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='cartApp']/div[@class='page-spinner page-spinner--out']/div[@class='large-view size-l']//div[@class='fluid-large-view']//section[@class='fluid-large-view__sidebar']//button[@type='button']")))
    element.send_keys(Keys.RETURN)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='fld-e']")))
    element.send_keys(EMAIL)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='fld-p1']")))
    element.send_keys(PASSWORD)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@class='cia-app-container']/div[@class='cia-actual-full-page-wrapper lv']/section/main[@class='cia-wrapper container']//form/div[3]/button")))
    element.send_keys(Keys.RETURN)
    driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='credit-card-cvv']"))))
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='credit-card-cvv']")))
    element.send_keys(CVV)
    driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='checkoutApp']/div[@class='page-spinner page-spinner--out']/div[1]/div[1]//div[@class='checkout__container checkout__container-fast-track']/div[@class='checkout__col checkout__col--primary']//div[@class='button--place-order-fast-track']/button"))))
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='checkoutApp']/div[@class='page-spinner page-spinner--out']/div[1]/div[1]//div[@class='checkout__container checkout__container-fast-track']/div[@class='checkout__col checkout__col--primary']//div[@class='button--place-order-fast-track']/button")))
    element.send_keys(Keys.RETURN)
    
try:
    runBestBuyBotLocal()
    message = client.messages \
        .create(
            body='Success!',
            from_=TWILIONUMBER,
            to=MYNUMBER
        )
except:
    message = client.messages \
        .create(
            body='Fail! ' + traceback.format_exc(),
            from_=TWILIONUMBER,
            to=MYNUMBER
        )

The following code is the basically the same process but adapted slightly to function on Heroku remote dyno. It is a part of my Heroku web app. The below code is NOT making the purchase successfully when run on Heroku remote dyno. It is instead throwing a Timeout error on the line element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@class='c-portal']/div[@role='dialog']/div[1]/div[@role='dialog']/div[@role='document']//a[@role='button']"))) This is really weird because I have the same line in runBestBuyBotLocal, and when runBestBuyBotLocal gets is executed locally, no timeout exception gets thrown on that line, and the purchase is made successfully. So, because of the timeout exception being thrown, every time I run the following code on a Heroku remote dyno, I get an SMS message from my Twilio phone number containing "Fail!":

def runBestBuyBotRemote():
    chrome_options = webdriver.ChromeOptions()
    chrome_options.binary_location = os.environ.get("GOOGLE_CHROME_BIN")
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_options.add_argument("--no-sandbox")
    driver = webdriver.Chrome(executable_path=os.environ.get("CHROMEDRIVER_PATH"), chrome_options=chrome_options)
    driver.get("https://www.bestbuy.com/site/spongebob-squarepants-mini-plush-styles-may-vary/6404213.p?skuId=6404213")
    wait = WebDriverWait(driver, 10)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html/body[@class='size-l']/div[@class='pl-page-content']//div[@class='container-v2']/div[@class='row v-m-bottom-g']/div[2]//div[@class='col-xs-12']/div[6]/div[@class='None']/div/div/div/button[@type='button']")))
    element.send_keys(Keys.RETURN)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@class='c-portal']/div[@role='dialog']/div[1]/div[@role='dialog']/div[@role='document']//a[@role='button']")))
    element.send_keys(Keys.RETURN)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='cartApp']/div[@class='page-spinner page-spinner--out']/div[@class='large-view size-l']//div[@class='fluid-large-view']//section[@class='fluid-large-view__sidebar']//button[@type='button']")))
    element.send_keys(Keys.RETURN)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='fld-e']")))
    element.send_keys(EMAIL)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='fld-p1']")))
    element.send_keys(PASSWORD)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@class='cia-app-container']/div[@class='cia-actual-full-page-wrapper lv']/section/main[@class='cia-wrapper container']//form/div[3]/button")))
    element.send_keys(Keys.RETURN)
    driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='credit-card-cvv']"))))
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='credit-card-cvv']")))
    element.send_keys(CVV)
    driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='checkoutApp']/div[@class='page-spinner page-spinner--out']/div[1]/div[1]//div[@class='checkout__container checkout__container-fast-track']/div[@class='checkout__col checkout__col--primary']//div[@class='button--place-order-fast-track']/button"))))
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='checkoutApp']/div[@class='page-spinner page-spinner--out']/div[1]/div[1]//div[@class='checkout__container checkout__container-fast-track']/div[@class='checkout__col checkout__col--primary']//div[@class='button--place-order-fast-track']/button")))
    element.send_keys(Keys.RETURN)

client = Client(ACCOUNT_SID, AUTH_TOKEN)
try:
    runBestBuyBotRemote()
    message = client.messages \
        .create(
            body='Success!',
            from_=TWILIONUMBER,
            to=MYNUMBER
        )
except:
    message = client.messages \
        .create(
            body='Fail!' + traceback.format_exc(),
            from_=TWILIONUMBER,
            to=MYNUMBER
        )

So, the question is: does anyone have an idea as to why runBestBuyBotLocal is NOT throwing a timeout exception when run locally and why runBestBuyBotRemote IS throwing a timeout exception when run on my Heroku remote dyno? In other words, why am I getting a "Success!" SMS message when the process is run locally and a "Fail!" message when it is run on my Heroku remote dyno?

I'm confused why the process is succeeding when run locally and failing when run on Heroku remote dyno. The only difference between runBestBuyBoyLocal and runBestBuyBotRemote functions is how the webdriver is instantiated. So the reason for one failing and one succeeding could be due to this difference, but I'm not sure.

Let me know if I need to clear up anything!

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
watersheep23
  • 339
  • 1
  • 3
  • 17

4 Answers4

2
import traceback

try:
    WebDriverWait(driver, 2).until(EC.element_to_be_clickable(
        (By.XPATH, '//b[@id = "country_england"]/preceding-sibling::input'))).click()
except:
    traceback.print_exc()

use traceback to print the full exception trace .

enter image description here

PDHide
  • 18,113
  • 2
  • 31
  • 46
  • Thank you! This works; traceback is definitely a nice tool to see the exact error message of the exception; I'm new to exception handling in python. Btw the module is now called traceback2, so I had to do "import traceback2 as traceback". Thanks again! – watersheep23 Jan 31 '21 at 04:17
  • But the main question still remains...why am I getting a Timeout Exception when I run remotely and why am I not getting one when I run locally? – watersheep23 Feb 03 '21 at 20:01
  • 1
    @watersheep23 - When you say timeout, what are you referring to? are you referring to the selenium exception for e.g., in the java world `org.openqa.selenium.TimeoutException` represents this when a wait for an element times out. This usually means that the element in question doesn't seem to appear at all when you run in the remote mode. Have you tried taking a screenshot of what the UI looks like when this failure happens. You should add logic to take a screenshot just before the operation and in your finally block as well to see what is happening on the UI. – Krishnan Mahadevan Feb 04 '21 at 04:13
  • Yes that is what I am referring to, for python it's selenium.common.exceptions.TimeoutException. The thing is though is that I'm running the code on a remote Heroku dyno, so I don't actually get to see the Chrome browser open on my computer when it tries to make a purchase. Therefore, I don't think that I would be able to take a screenshot. – watersheep23 Feb 04 '21 at 16:09
  • @PDHide yes the trace just said selenium.common.exceptions.TimeoutExceptions on the line I mentioned in the question. – watersheep23 Feb 04 '21 at 16:10
  • @watersheep23 take screenshot of the website and see whats happening in the remote system – PDHide Feb 04 '21 at 16:11
  • The thing is is that my web app literally has no UI lol, it's kind of unnecessary to have one for my purposes. But are you saying that I can still take a screenshot even though it's running on the remote system? – watersheep23 Feb 04 '21 at 16:12
  • did you assign the env variable for heroku config? – Alin Stelian Feb 05 '21 at 12:24
  • @AlinStelian yes – watersheep23 Feb 08 '21 at 19:42
1

In my opinion, you should try:

  1. Set a screen resolution: chrome_options.add_argument('window-size=1920x1080'); Websites can have different layout on different resolutions...
  2. Not use full XPaths, for example:
"/html//div[@class='c-portal']/div[@role='dialog']/div[1]/div[@role='dialog']/div[@role='document']//a[@role='button']"

Instead use: "//a[@role='button']", if multiple elements exist in DOM with this property, try to expand condition i.e. "//a[@role='button' and contains(@id, 'foo')]" Look: How to Find XPath?

Alag
  • 83
  • 4
1

You need to consider a couple of things as follows:

You can find a detailed discussion in Not able to maximize Chrome Window in headless mode.

  • If your test step is to interact with the element instead of presence_of_element_located() you need to induce WebDriverWait for element_to_be_clickable().

    • Example:

      WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html/body[@class='size-l']/div[@class='pl-page-content']//div[@class='container-v2']/div[@class='row v-m-bottom-g']/div[2]//div[@class='col-xs-12']/div[6]/div[@class='None']/div/div/div/button[@type='button']"))).send_keys(Keys.RETURN)

You can find a detailed discussion in How to insert a value in a field Text using Selenium?.

  • Absolute xpath makes your tests brittle. You may like to replace them with relative xpath. As an example, replace:

    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='cartApp']/div[@class='page-spinner page-spinner--out']/div[@class='large-view size-l']//div[@class='fluid-large-view']//section[@class='fluid-large-view__sidebar']//button[@type='button']")))

    with:

    element = wait.until(EC.presence_of_element_located((By.XPATH, "//button[@type='button'][@attribute2='value'][@attribute3='value']")))
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • 1
    Why exactly does the window size need to be maximized? When I run locally, the window size is not maximized, and the webdriver process works fine. – watersheep23 Feb 12 '21 at 06:09
1

All the answers above helped me figure it out. It was all about the flags. Assuming you have the build packs installed, and config vars added (many other answers already mention these), then you may still need the correct flags added as options.

Even thought the Chrome Buildpack says it adds the flags automatically, for me, it did not. I also needed others. If you are scraping a page you need a window size, else you might see unknown error: session deleted because of page crash , and apparently there are sizes issues re: heroku v chrome (or something) here

chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument("--disable-gpu")
## might not be needed
chrome_options.add_argument("--remote-debugging-port=9222")
chrome_options.add_argument('--window-size=1920x1480')
Mote Zart
  • 861
  • 3
  • 17
  • 33