Python Selenium Web Scraping the data from the charts

Question

I am trying to web-scrape the chart data from this page https://www.focusireland.ie/resource-hub/latest-figures-homelessness-ireland/

On the right top, there is a chart which details Number of homeless people in Ireland.

I am trying to web-scrape the axis which is Jul - 14 and it's corresponding value 3258. But, I couldn't do locate those element and it's value using x-path.

I tried something like this to web-scrape the value

Val1 = driver.find_element_by_xpath("//*[@id='chart']/div[7]").text

& it didn't work.

And tried this:-

element_to_hover_over = driver.find_element_by_xpath("//div[@class='label series smaller' | @class='label series smaller hover']/span[text()='Jul-14'").text
print(element_to_hover_over)

It didn't work either for me and returned an error message as

Unable to locate an element with the xpath expression //div[@class='label series smaller' | @class='label series smaller hover']/span[text()='Jul-14' because of the following error:
SyntaxError: Failed to execute 'evaluate' on 'Document': The string '//div[@class='label series smaller' | @class='label series smaller hover']/span[text()='Jul-14'' is not a valid XPath expression.

score 1 · Answer 1 · answered Apr 23 '18 at 14:11

1

Tha data are not on that website. You have to parse source iframe of the chart ( datawrapper.dwcdn.net/** ). Then you have to extract and parse JS variable (__dw.params.data).

This is not so hard, see this.

answered Apr 23 '18 at 14:11

Nestor Yanchuk

1,186
8
10

:- Can you show that for one of the axis please. I am finding it really hard to understand this with no background to JS. – Student of the Digital World Apr 23 '18 at 14:13
Look at the link I've provided, then just run `wd.execute_script("return __dw.params.data")`. It is going to be an easy-to-parse string. – Nestor Yanchuk Apr 24 '18 at 11:25

Ankur Singh · Answer 2 · 2018-04-23T14:48:14.457

1

You can try below code to get text :

element_to_hover_over = firefox.find_element_by_xpath("//span[text()='Jul-14']")

hover = ActionChains(firefox).move_to_element(element_to_hover_over)
hover.perform()

element_to_hover_over = firefox.find_element_by_xpath("//div[@class='label value outline  showOnHover hover']/span")

element_to_hover_over.text

Your second chart is inside frame first you need to switch into frame :-

frame :- datawrapper-chart-8ZOKk

then try below xpath to get value in list

list <webElement> = (//div[@class='label series smaller' and @data-column='X.1'])[1]/span

then iterate through every row and get value

edited Apr 23 '18 at 14:48

answered Apr 23 '18 at 14:20

Ankur Singh

1,239
1
8
18

this would get me the value of 3258. But getting the value of Jul-14 is still a task here. – Student of the Digital World Apr 23 '18 at 14:22
you can element_to_hover_over .text to get value – Ankur Singh Apr 23 '18 at 14:25
try this :- //div[@class='label series smaller' | @class='label series smaller hover']/span[text()="Jul-14"] – Ankur Singh Apr 23 '18 at 14:31
Sorry, but I think you didn't get my point. I am looking to go back from chart to table. So, I would like to write a code that is dynamic - here we are putting in the variable as Jul-14 in element_to_hover_over – Student of the Digital World Apr 23 '18 at 14:33
Updated my answer – Ankur Singh Apr 23 '18 at 14:48
This is so difficult for me to grasp. I really don't know how to use your code in my program. The code is returning error. I am going to go back to my basics and read them. I am not an expert in this subject. I require those data for some analysis purposes. If you can, rephrase your code in a way that it would run smooth if I copy and paste that will be helpful. I feel bad to ask this again. – Student of the Digital World Apr 23 '18 at 14:57
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/169600/discussion-between-sid29-and-ankur-singh). – Student of the Digital World Apr 23 '18 at 15:00

Python Selenium Web Scraping the data from the charts

2 Answers2

element_to_hover_over.text