Python code works locally but not on server (Discord bot with selenium)

Question

My code takes user input and uses it to search for an item on a webpage. it scrapes the html code of the webpage in order to get the items that appeared as a result:

driver = webdriver.Chrome()
driver.get(urlItem)
#this last line is the one that's not working on heroku
item_list = driver.find_elements_by_class_name('price-history-link')

this is the html code:

<ul class="no-padding small-block-grid-3 large-block-grid-5 text-center word-wrap-break-word">
<li>
<a href="/item/7399/"><img src="/assets/imgs/items/7399.gif" alt="A Grey Faerie Doll" title="A Grey Faerie Doll" class="item-result-image"></a><br><a href="/item/7399/">A Grey Faerie Doll</a>
<br><span class="text-small"><a href="/item/7399/price-history/" class="price-history-link" title="October 31, 2019">2,400,000 NP</a></span>
</li>
</ul>

When I run the code locally, the bot works perfectly. When I run the exact same code from my hosting site (heroku), it doesn't return any items. I thought that maybe it was a loading issue, so I added a :

WebDriverWait(driver, 10).until(ec.visibility_of_element_located((By.CLASS_NAME, "price-history-link")))

when this code is in place, it returns an exception (only when run on the hosting site, it works perfectly when run on my pc)

raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message:

yes, the exception ends there, it doesnt actually show me the message.

I really don't understand why it won't work on heroku... any help is very appreciated!

Edit: Put in a catch for a NoSuchElementException, but it didn't catch anything

Answer 1

You saw it right. WebDriverWait() in conjunction with expected_conditions() will always return TimeoutException on failure.

You can find a detailed discussion in Random Selenium E2e Tests Fail because of timeouts on Azure DevOps but work locally and with remote Selenium (BrowserStack Automate)

---

This usecase

As you are using the Locator Strategy CLASS_NAME as price-history-link possibly this CLASS_NAME is associated with some hidden elements as well. Hence, when you use find_elements_by_class_name(), elements are identified where as visibility_of_element_located() results into TimeoutException.

---

Solution

To simulate find_elements_by_class_name() along with WebDriverWait() and in conjunction with expected_conditions() you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following solutions:

• Using CSS_SELECTOR:

  item_list = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.price-history-link[href$='/price-history/'][title*='20']")))
  

• Using XPATH:

  item_list = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[@class='price-history-link' and contains(@href, '/price-history/')][contains(., 'NP')]")))
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC