using Selenium to get texts inside 'ul' tag?

Question

Please help me to find the solution to get the text inside 'ul' tag.

I want to get the information which is separated with commas like: 'Contains Enzymatically Active B-Vitamins, Dietary Supplement, Non-GMO LE Certified'

website link: https://ca.iherb.com/pr/Life-Extension-BioActive-Complete-B-Complex-60-Vegetarian-Capsules/67051

picture: enter image description here

This is the HTML code:

<ul>
  <li>Contains Enzymatically Active B-Vitamins
  </li>
  <li>Dietary Supplement
  </li>
  <li>Non-GMO LE Certified
  </li>
</ul>

get text from all `li` and use `",".join(list_of_li)` – furas Dec 28 '20 at 05:44 — furas, Dec 28 '20 at 05:44

score 1 · Accepted Answer · answered Dec 28 '20 at 06:36

This should do it:

from selenium import webdriver

link = 'https://ca.iherb.com/pr/Life-Extension-BioActive-Complete-B-Complex-60-Vegetarian-Capsules/67051'

with webdriver.Chrome() as driver:
    driver.get(link)
    elements = ', '.join([item.text for item in driver.find_elements_by_css_selector("[itemprop='description'] > ul:nth-of-type(1) > li")])
    print(elements)

Output:

Contains Enzymatically Active B-Vitamins, Dietary Supplement, Non-GMO LE Certified

undetected Selenium · Answer 2 · 2020-12-28T07:12:43.643

To extract the texts e.g. Contains Enzymatically Active B-Vitamins, Dietary Supplement using Selenium and python you can use either of the following Locator Strategies:

Using CSS_SELECTOR and printing the list:

driver.get('https://ca.iherb.com/pr/Life-Extension-BioActive-Complete-B-Complex-60-Vegetarian-Capsules/67051')
print([my_elem.text for my_elem in driver.find_elements_by_css_selector("div[itemprop='description']>ul li")])

Console Output:

['Contains Enzymatically Active B-Vitamins', 'Dietary Supplement', 'Non-GMO LE Certified ', 'Promotes healthy metabolism of glucose, fat & alcohol', 'Supports the healthy energy production your body needs', 'Encourages healthy organ function, cognitive health & more', 'Helps inhibit potential vitamin B deficiency']

Using XPATH and printing the elements in a comma seperated string:

driver.get('https://ca.iherb.com/pr/Life-Extension-BioActive-Complete-B-Complex-60-Vegetarian-Capsules/67051')
print(', '.join([my_elem.text for my_elem in driver.find_elements_by_xpath("//div[@itemprop='description']/ul//li")]))

Console Output:

Contains Enzymatically Active B-Vitamins, Dietary Supplement, Non-GMO LE Certified , Promotes healthy metabolism of glucose, fat & alcohol, Supports the healthy energy production your body needs, Encourages healthy organ function, cognitive health & more, Helps inhibit potential vitamin B deficiency

To extract the texts e.g. Contains Enzymatically Active B-Vitamins, Dietary Supplement ideally you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR and printing the list:

driver.get('https://ca.iherb.com/pr/Life-Extension-BioActive-Complete-B-Complex-60-Vegetarian-Capsules/67051')
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div[itemprop='description']>ul li")))])

Console Output:

['Contains Enzymatically Active B-Vitamins', 'Dietary Supplement', 'Non-GMO LE Certified ', 'Promotes healthy metabolism of glucose, fat & alcohol', 'Supports the healthy energy production your body needs', 'Encourages healthy organ function, cognitive health & more', 'Helps inhibit potential vitamin B deficiency']

Using XPATH and printing the elements in a comma seperated string:

driver.get('https://ca.iherb.com/pr/Life-Extension-BioActive-Complete-B-Complex-60-Vegetarian-Capsules/67051')
print(', '.join([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@itemprop='description']/ul//li")))]))

Console Output:

Contains Enzymatically Active B-Vitamins, Dietary Supplement, Non-GMO LE Certified , Promotes healthy metabolism of glucose, fat & alcohol, Supports the healthy energy production your body needs, Encourages healthy organ function, cognitive health & more, Helps inhibit potential vitamin B deficiency

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

score 0 · Answer 3 · answered Dec 28 '20 at 05:55

You can always get all elements li, get text from all these elements and use ", ".join(elements)

Code for your small example

text = '''
<ul>
  <li>Contains Enzymatically Active B-Vitamins
  </li>
  <li>Dietary Supplement
  </li>
  <li>Non-GMO LE Certified
  </li>
</ul>'''

import selenium.webdriver

driver = selenium.webdriver.Firefox()

driver.get("data:text/html;charset=utf-8," + text)

elements = driver.find_elements_by_tag_name('li')

elements = [i.text for i in elements]

print(", ".join(elements))

score 0 · Answer 4 · answered Dec 28 '20 at 06:16

from selenium import webdriver
from shutil import which
chrome_path = which('chromedriver.exe')

driver = webdriver.Chrome(executable_path = chrome_path)

li_eliments = driver.find_elements_by_tag_name('li')

elements = []
for e in li_eliments.text:
    elements.append(e)
print(", ".join(elements))

score 0 · Answer 5 · answered Dec 28 '20 at 06:44

Alrighty so Selenium is for web automation, but data scraping (like what it seems like you're trying to do) is more for requests, and beautiful soup. There have been posts about using Selenium that work, but it is easier to use these so you don't have to launch a web browser to do it like for selenium.

r = requests.get("https://ca.iherb.com/pr/Life-Extension-BioActive-Complete-B-Complex-60-Vegetarian-Capsules/67051")
soup = BeautifulSoup(r.content, 'html.parser')
list_items = soup.find('div', itemprop="description")
found = str(re.findall(r'itemprop="description"><ul><li>(\D+)', str(list_items)))

This take just a second whereas the other methods can take much longer to load a browser and navigate to the website for this information. Once you get this information and use regex to find the appropriate tag, you can use regex to clean it up just the text.

newfound = re.sub(r"</li>|[\[']", '', found)
newfound2 = re.sub(r"<li>", ', ', newfound)
stripped = newfound2.split('\\xa0', 1)[0]

The lines itemprop="description"><ul><li> and \xa0 are both from viewing the source of the page and finding the list element there. Here's some info on regex: https://www.guru99.com/python-regular-expressions-complete-tutorial.html

using Selenium to get texts inside 'ul' tag?

5 Answers5