Print all values in a column from table Selenium

Question

I want to return all the values in the MeSH column at this link: https://ii.nlm.nih.gov/cgi-bin/II/Interactive/checkPubMed.pl. The link doesn't store the search results so to get to the actual link, go here https://ii.nlm.nih.gov/Interactive/MTI/phrase2mesh.shtml and search "baculovirus" and press submit.

driver.get('https://ii.nlm.nih.gov/Interactive/MTI/phrase2mesh.shtml')
#excluded code here but basically input "baculovirus" into the search bar and search
#brings you to this link https://ii.nlm.nih.gov/cgi-bin/II/Interactive/checkPubMed.pl
table = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.TAG_NAME,"table")))
        for row in table(3): #to get the third column of the table
            print(row.text)

Not sure how to move forward, any help would be appreciated!

Hi there what would you expected output be? – Arundeep Chohan Sep 24 '20 at 23:30 — Arundeep Chohan, Sep 24 '20 at 23:30

Arundeep Chohan · Answer 1 · 2020-09-25T00:57:17.223

0

You could simply do the following.

table = WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.XPATH,"//table[2]/tbody/tr/td[3]")))
for td in table: #to get the third column of the table
    print(td.text)

edited Sep 25 '20 at 00:57

answered Sep 24 '20 at 23:34

Arundeep Chohan

9,779
5
15
32

there's an issue in the ```for td in table:```, the webelement is not iterable – Brian Guan Sep 25 '20 at 00:51
@Brian Guan oh my bad i forgot to change the presence – Arundeep Chohan Sep 25 '20 at 00:56

undetected Selenium · Answer 2 · 2020-09-25T07:18:04.870

To print all the values from the MeSH Term column you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR and text attribute:

driver.get('https://ii.nlm.nih.gov/Interactive/MTI/phrase2mesh.shtml')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "textarea[name='InputText']"))).send_keys("baculovirus")
driver.find_element_by_css_selector("input[value='Submit Phrase2MeSH Request']").click()
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "body table:nth-of-type(2) tr td:nth-child(3)")))])

Using XPATH and get_attribute():

driver.get('https://ii.nlm.nih.gov/Interactive/MTI/phrase2mesh.shtml')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//textarea[@name='InputText']"))).send_keys("baculovirus")
driver.find_element_by_xpath("//input[@value='Submit Phrase2MeSH Request']").click()
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//b[starts-with(., 'Descriptors')]//following::table[1]//tr//td[not(contains(@align,'right'))]")))])

Console Output:

['Baculoviridae', 'Recombinant Proteins', 'Genetic Vectors', 'Sf9 Cells', 'Nucleopolyhedroviruses', 'Cell Line', 'Spodoptera']

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

References

You can find a couple of relevant discussions on NoSuchElementException in:

Print all values in a column from table Selenium

2 Answers2

References