Problem with scraping in Selenium. It does not scrape

Question

I would like to get the number of rounds (14, 15, 16, 17, etc.), then just scrape the numbers, no ".ROUND". I wrote a basic version and a better version, but they don't work. I don't get errors, but I get >>>> (without any text). How to do?

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 20)

Values_PremierLeague = []

driver.get("https://www.betexplorer.com/soccer/england/premier-league/fixtures/")

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "th.h-text-left")))

for PremierLeague in driver.find_elements(By.CLASS_NAME, "th.h-text-left"):
    PremierLeague_text = PremierLeague.text
    Values_PremierLeague.append(tuple([PremierLeague_text]))
    print(PremierLeague_text)
driver.close

or

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import re

wait = WebDriverWait(driver, 20)

Values_PremierLeague = []

driver.get("https://www.betexplorer.com/soccer/england/premier-league/fixtures/")

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".table-main__datetime")))

for PremierLeague in driver.find_elements(By.XPATH, "//*[@class='h-text-left' and contains(text(),'Round')]"):
    PremierLeague_text = PremierLeague.text
    number = re.findall(r'\d+', s)
    Values_PremierLeague.append(tuple([number]))
    print(number)
driver.close

UPDATE

CREATE TABLE "BASE_Giornate" (
    "ID_Round"  INTEGER,
    "Number_Round"  INTEGER,
    "Id_Tournment"  INTEGER,
    PRIMARY KEY("ID_Giornata" AUTOINCREMENT)
);

and the code already working

sqlite_insert_query_PremierLeague = 'INSERT INTO BASE_Giornate (Number_Round) VALUES (?);'
cursor.executemany(sqlite_insert_query_PremierLeague, Values_PremierLeague)
count_squadre_PremierLeague = cursor.rowcount
con.commit()

So the first one you accidentally used class name instead of css selector. s should be premier_League_text in the second. — Arundeep Chohan, Nov 23 '21 at 20:09
@ArundeepChohan Yes. The first one gets the class name wrong and in any case doesn't have the setting to just scrape the number by eliminating ".ROUND". The second one should work, but there is something wrong. Can you help me? Thanks — Erling Olsen, Nov 23 '21 at 20:16

score 1 · Accepted Answer · answered Nov 23 '21 at 20:16

1

driver.get("https://www.betexplorer.com/soccer/england/premier-league/fixtures/")

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".table-main__datetime")))

for PremierLeague in driver.find_elements(By.XPATH, "//*[@class='h-text-left' and contains(text(),'Round')]"):
    PremierLeague_text = PremierLeague.text
    number = re.findall(r'\d+', PremierLeague_text)
    Values_PremierLeague.append(tuple([number]))
    print(number)

Not sure what you meant by numbers also not sure why you want tuples. But you had an issue where you never used PremierLeague_text

Outputs:

['13']
['14']
['15']
['16']
['17']
['18']
['19']
['20']
['21']
['22']
['23']
['24']
['25']
['26']
['27']
['28']
['29']
['30']
['31']
['32']
['33']
['34']
['35']
['36']
['37']
['38']

answered Nov 23 '21 at 20:16

Arundeep Chohan

9,779
5
15
32

That's exactly what I wanted to achieve. Thanks. By numbers I meant this. As for the tuples, I have to save these numbers in a column of a database, and I have to scrape for other different tournaments as well. At this point you make me doubt: are tuples not necessary? – Erling Olsen Nov 23 '21 at 20:17
If you don't have multiple values in it it's best to just do an array. – Arundeep Chohan Nov 23 '21 at 20:26
Ah OK. It is only 1 value. What should I do with the array? (if you want you can not answer). I'll vote for your answer anyway. Thanks – Erling Olsen Nov 23 '21 at 20:50
Excuse me. I didn't know that if it was closed you couldn't answer. I reopened it momentarily (then I will close it again). Excuse me and thank you – Erling Olsen Nov 23 '21 at 21:24
Oh not sure how your database is set up so cant really comment – Arundeep Chohan Nov 23 '21 at 21:34
I updated the question by showing you my small database. I would like to save those round numbers without the square bracket [] and apostrophe '. can you help me? Thanks – Erling Olsen Nov 23 '21 at 23:32
You can probably just loop it and then fstring it so it gets the value on loop. – Arundeep Chohan Nov 23 '21 at 23:43
How can I tighten it so that it gets the looped value? Can you show me the code in your answer please? I'm just starting out with Python. Thanks :) – Erling Olsen Nov 23 '21 at 23:45
https://stackoverflow.com/questions/51312952/python-sqlite3-using-f-strings-for-update-database-function Theres a ton of examples for fstrings for querying. – Arundeep Chohan Nov 23 '21 at 23:54

Problem with scraping in Selenium. It does not scrape

1 Answers1