Scraping data from high charts

Question

I have two problems: first, I cannot click on the show all bottom; and, second, I cannot get the data from the high chart.

I saw some examples for the high chart on Stack Overflow; however, I did not get how people figure our which JS code to execute.

I tried the following code to achieve that:

from selenium import webdriver
DRIVER_PATH = r"C:\Users\XX\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
url = "https://siterankdata.com/wsj.com"
driver.get(url)
driver.find_element_by_xpath('//*[@id="smallchart"]/div/div/svg/g[17]/g/text/tspan').click() # Does not work I try to click on the show all button.

I would appreciate any help!

Which is that element `//*[@id="smallchart"]/div/div/svg/g[17]/g/text/tspan` which you are trying to click? — undetected Selenium, Nov 18 '20 at 17:21
Hi @Dogukan Yılmaz, You can use JS global variable `Highcharts` to get acces to charts and execute for example: `Highcharts.charts[0].series[x].points` for data points. — ppotaczek, Nov 19 '20 at 08:10
Hi @Ppotaczek, I think the data is not loaded with a high chart but with AmCharts, do you have any idea about how to get that? — Dogukan Yılmaz, Nov 19 '20 at 22:23

score 1 · Accepted Answer · answered Nov 18 '20 at 17:55

1

To click on show all button Use WebDriverWait() and wait for element_to_be_clickable() and following xpath

driver.get(url)
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//div[@id='smallchart']//*[name()='svg']/*[name()='g'][17]/*[name()='g']/*[name()='text']/*[name()='tspan']"))).click()

You need to import below libraries

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

answered Nov 18 '20 at 17:55

KunduK

32,888
5
17
41

@DogukanYılmaz : what data are you expecting from high chart? – KunduK Nov 18 '20 at 19:12
The values from Website Ranking Data with dates and values – Dogukan Yılmaz Nov 18 '20 at 19:44

undetected Selenium · Answer 2 · 2020-11-18T20:15:50.307

1

The element with text as Show all is a svg element so to click on the element you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

driver.get("https://siterankdata.com/wsj.com")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#smallchart svg g text[text-anchor='start'] tspan"))).click()

Using XPATH:

driver.get("https://siterankdata.com/wsj.com")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@id="smallchart"]//*[name()='svg']//*[name()='g']//*[name()='text']//*[name()='tspan' and text()='Show all']"))).click()

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

References

You can find a couple of relevant discussions on interacting with SVG element in:

edited Nov 18 '20 at 20:15

answered Nov 18 '20 at 20:07

undetected Selenium

183,867
41
278
352

1

Thank you and how about getting the data from the chart later? – Dogukan Yılmaz Nov 18 '20 at 20:44
@DogukanYılmaz Which data are you specifically looking at? – undetected Selenium Nov 18 '20 at 20:49
Date and ranking values from the same table that I click on the "Show All" – Dogukan Yılmaz Nov 18 '20 at 20:52
@DogukanYılmaz But those datas doesn't appears at a time. You have to _Mouse Hover_ around using `ActionChains` and keep on scraping the data. Check [this discussion](https://stackoverflow.com/questions/57901045/scrape-website-with-dynamic-mouseover-event/57903003#57903003). – undetected Selenium Nov 18 '20 at 20:56
1

Ok let me try to understand what was there and implement for my own – Dogukan Yılmaz Nov 18 '20 at 20:57
Well actually how to scrape data from the chart was a part of my question, so if I cannot figure that, I would still need help. I will accept once I get the data – Dogukan Yılmaz Nov 18 '20 at 21:05

Scraping data from high charts

2 Answers2

References