3

I am trying to scrape the data from http://fuelinsights.gasbuddy.com/Charts using Python and Selenium. The difficult part is that the data only appear when a point on the line graph is hovered over. Currently, my issue is an inability to create a list of all the hover over objects. My code so far is below:

from selenium import webdriver as web
from selenium.webdriver.common.action_chains import ActionChains

driver = web.Chrome('driver path')

driver.get('http://fuelinsights.gasbuddy.com/Charts')

test= driver.find_elements_by_xpath('//*[@class="highcharts-markers"]')

print(test)

`

which gives me test=[]. Previously, I have used beautifulsoup for all of my scraping projects, but I have redone some of my previous projects to make sure that I understand how Selenium works and haven't had issues.

If anyone can help me solve this issue so I can create a list of the items that I can use ActionChains to hover over and extract the price and date from it would be much appreciated.

Thank you!

****EDIT**** To clarify, I have looked over numerous other posts concerning SVG and g elements and Highcharts, but I am still short on a solution to this problem. I have tried numerous Xpaths (and other find_elements_by options), but have only been able to come to two results: (1) the Xpath is valid, but does not contain any elements, or (2) InvalidSelectorException indicating that I was unable to locate an element with the xpath expression. I believe this comes down to simply incorrectly specifying my Xpath, but I am at a loss for how to find the correct Xpath.

Let's Boogie
  • 53
  • 2
  • 6
  • The `highcharts-markers` class is within a `` tag which is within a `svg` tag. Possible duplicate of [Selenium WebDriver \[Java\]: How to Click on elements within an SVG using XPath](https://stackoverflow.com/questions/41829000/selenium-webdriver-java-how-to-click-on-elements-within-an-svg-using-xpath) – undetected Selenium Jan 10 '18 at 07:55
  • 1
    Thanks DebanjanB this has provided some useful insight. These are very similar questions, but I have been unable to solve my problem using the proposed solutions provided in the link (admittedly this could be due to my own shortcomings converting from Java to Python though). – Let's Boogie Jan 10 '18 at 21:23

2 Answers2

5

You can't use the Xpath which you have mentioned above for locating the elements inside the svg tag.

Xpath which you can use to create a list of hover objects is:

//*[name()='svg']//*[name()='g' and @class='highcharts-markers']/*[name()='path']

I have written a java program for getting the text of all the tool-tip elements. You can use the logic and write a corresponding python code:

1. Get List of tooltip Elements

 List <WebElement> highChartElements= driver.findElements(By.xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-markers']/*[name()='path']"));

2. Iterate through the list and use action class for moving and clicking on all the tooltip Elements

3. Get the text of the tooltip elements.

for(WebElement element:highChartElements){
        Actions action = new Actions(driver);
        action.moveToElement(element).click().perform();
        Thread.sleep(3000);
        List<WebElement> highChartToolTipTextElements= driver.findElements(By.xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-tooltip']/*[name()='text']/*[name()='tspan']"));
        for(WebElement toolTipElement:highChartToolTipTextElements){
            System.out.println("The text for the elements is"+toolTipElement.getText());
        }
    }
cruisepandey
  • 28,520
  • 6
  • 20
  • 38
Monika
  • 714
  • 1
  • 4
  • 10
  • Monika, thanks for the pointers. I had to tweak the Xpath you provided slightly for it to run, but I still am getting an empty list back from the driver.find_elements_by_xpath command. My Xpath is now '//*[name()="svg"]//*[name()="g"]//*[@class="highcharts-markers"]/*[name="path"]' do you have any other advice on how to fix this? – Let's Boogie Jan 10 '18 at 19:19
  • Don't change the xpath, use the one I have provided. The xpath should work, I have executed the program on my machine and it worked.Why did you tweak the Xpath? – Monika Jan 11 '18 at 05:12
  • The Xpath you specified when used in python in the following command `test= driver.find_elements_by_xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-markers']/*[name()='path']")` returns test=[]. I'm not sure why this provides you with results, but returns nothing to me. Any further guidance is much appreciated! – Let's Boogie Jan 11 '18 at 16:33
3

Thanks You ! 2 years later, i'm facing to an equivalent project and used your example to learn how to get the job done with python & Firefox. Perhaps the following code will be useful to some people.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

driver = webdriver.Firefox(executable_path=r'path\to\the\geckodriver.exe')
driver.get('http://fuelinsights.gasbuddy.com/Charts')
element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "highcharts-markers")))
test = driver.find_elements_by_xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-markers']/*[name()='path']")

res = []
for el in test:
    hover = ActionChains(driver).move_to_element(el)
    hover.perform()
    date = driver.find_elements_by_css_selector(".highcharts-tooltip > text:nth-child(5) > tspan:nth-child(1)")
    price = driver.find_elements_by_css_selector(".highcharts-tooltip > text:nth-child(5) > tspan:nth-child(4)")
    res.append((date[0].text, price[0].text))

"res" contains :

('Saturday, May 30, 2020', '1.978 $/gal')
('Friday, May 29, 2020', '1.979 $/gal')
('Thursday, May 28, 2020', '1.977 $/gal')
('Wednesday, May 27, 2020', '1.972 $/gal')
('Tuesday, May 26, 2020', '1.965 $/gal')
.......
ce.teuf
  • 746
  • 6
  • 13