I am trying to scrape data from robots.txt. But when it comes to xpath selectors they doesn't work. For example i am trying to select xpath by response.xpath('/urlset/url[1]/loc').get() , but there is no result. In firefox and in scrapy shell the source code corresponds(in chrome no). The robots.txt file that i am trying to scrape is sciencopen
Asked
Active
Viewed 19 times
0
Konstantin Pliev
- 1
- 1
-
Do you need the text from the `
` tag?. If yes, then try `response.xpath('/urlset/url[1]/loc/text').get()` – Krupal Vaghasiya Oct 25 '21 at 10:53 -
Please provide enough code so others can better understand or reproduce the problem. – Community Oct 25 '21 at 11:01
-
@KrupalVaghasiya: That won't work due to the default namespace. See duplicate link for how to deal with XML namespaces in XPath. – kjhughes Oct 25 '21 at 12:42