Using xpath gives empty output

Question

I want to get address but they provide me empty what I am doing wrong in the XPath..... these is the page link https://www.findtruckservice.com/page/cummins-sales-and-service-farmington-nm-430653

Snapshot of the address:

Code trials:

import scrapy
from scrapy import Selector
from scrapy_selenium import SeleniumRequest
from scrapy.http import Request


class TestSpider(scrapy.Spider):
    name = 'test'

    
    
    def start_requests(self):
            yield SeleniumRequest(
                url ="https://www.findtruckservice.com/search/?city=Florida%2C+CO&mainCat=1&subCat=Truck+Repair&lat=37.0731&lon=-106.247&cat_field=Mobile+Repair+-+Truck+Repair",
                wait_time = 3,
                screenshot = True,
                callback = self.parse,
                dont_filter = True
                )
    
    def parse(self, response):
            books = response.xpath("//h3//a//@href").extract()
            for book in books:
                url = response.urljoin(book)
                yield Request(url, callback=self.parse_book)
            
                    
    def parse_book(self, response):
            address=response.xpath("//div[1][@class='threecol align_left card']//div//text()").get()
            yield{
                'address':address
            }

undetected Selenium · Answer 1 · 2023-01-23T21:01:56.673

To print the desired text from the website you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:

Using XPATH and text attribute:

driver.get("https://www.findtruckservice.com/page/cummins-sales-and-service-farmington-nm-430653")
print(WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, "//h4[@class='sec-title' and text()='CONTACT']//following::div[@class='container']"))).text)

Using XPATH and get_attribute("textContent"):

driver.get("https://www.findtruckservice.com/page/cummins-sales-and-service-farmington-nm-430653")
print(WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, "//h4[@class='sec-title' and text()='CONTACT']//following::div[@class='container']"))).get_attribute("textContent"))

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Console Output:

Cummins Sales and Service
1101 N Troy King Rd
Farmington, NM
505-327-7331 (primary)
505-326-2948 (fax)

References

Link to useful documentation:

get_attribute() method Gets the given attribute or property of the element.
text attribute returns The text of the element.
Difference between text and innerHTML using Selenium

also they give me other detail of Hours,details.......I need only contact detail — Amen Aziz, Jan 23 '23 at 20:57

score 0 · Accepted Answer · answered Jan 23 '23 at 22:34

0

Try the following:

[...]

address = ' '.join([x.strip() for x in response.xpath("//div[@class='threecol align_left card'][1]/div[@class='container']/text()").extract()])

answered Jan 23 '23 at 22:34

Barry the Platipus

9,594
2
6
30

Using xpath gives empty output

2 Answers2

References