1

As my title said. To avoid deprecated warning of Selenium.I try to use the new format as I found. and somehow result return only 1 element. How can I fix it to return elements? FYI I scraping website. I'm working on Colab so I can share with my workmates easier.

Here my code

#!pip install selenium
#!pip install IPython
#!pip install pandas
#!apt-get update
#!apt install chromium-chromedriver

from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)

import pandas as pd
from IPython.display import display
from selenium.webdriver.common.by import By
website = 'https://www.bitkub.com/fee/cryptocurrency'
driver.get(website)

#pull_data
coins = driver.find_elements(By.TAG_NAME,'tr')

checking what element it contains (which somehow got only 1)

print(coins)
[<selenium.webdriver.remote.webelement.WebElement (session="091bac0ea46529043f4d84786ebc705f", element="76396e66-6516-4dd0-9d85-bc66bfad4676")>]
Visushi_Tiramitsu
  • 39
  • 6

2 Answers2

0

What is occuring is that you need webdriver waits and wait for visibility for all elements.

wait = WebDriverWait(driver, 10)
trs = wait.until(ec.visibility_of_all_elements_located((By.TAG_NAME, "tr")))

Import:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
Arundeep Chohan
  • 9,779
  • 5
  • 15
  • 32
0

Ideally coins should have been a list of elements.

However, to extract the name of the coins you need to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    driver.get('https://www.bitkub.com/fee/cryptocurrency')
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "tbody tr td:nth-child(2) span")))])

  • Using XPATH:

    driver.get('https://www.bitkub.com/fee/cryptocurrency')
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//following::td[2]//span")))])

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • Thanks, your method so so helpful. but can I use pandas to make Dataframe out of it? or Do I have to create a dictionary or something to sort it first? – Visushi_Tiramitsu Feb 10 '22 at 21:54
  • 1
    @Visushi_Tiramitsu Ofcoarse you can make a Dataframe out of it. See [this](https://stackoverflow.com/a/70160219/7429447) discussion. – undetected Selenium Feb 10 '22 at 22:02