following siblings in xpath returns empty result

Question

I have this html node

<li>
    <em>Description
    </em>
    <br>
    TEXT TEXT                
</li>

I want to extract the Text Text

I tried this:

 sel.xpath('//em[normalize-space(.) = "Description"]/following-sibling::*')

I got empty result.

Why please?

score 3 · Accepted Answer · answered Feb 06 '14 at 23:13

3

I found the solution myself and it is

'//li[contains(em,"Description")]/text()[last()]'

answered Feb 06 '14 at 23:13

Marco Dinatsoli

1

Was about to suggest that. Glad to see your XPath skills improve Marco :) – paul trmbrth Feb 07 '14 at 00:05
@pault. I always appreciate your efforts and help to me and your support. thank you a lot – Marco Dinatsoli Feb 07 '14 at 06:57
@pault. hello my friend, would you help me please http://stackoverflow.com/questions/21628891/scrapy-spider-returns-value-to-another-spider – Marco Dinatsoli Feb 07 '14 at 13:56
@paultrmbrth he my old friend, could you help me here please? http://stackoverflow.com/questions/25353650/scrapy-how-to-import-the-settings-to-override-it i need help – Marco Dinatsoli Aug 17 '14 at 21:22

helderdarocha · Answer 2 · 2014-02-06T23:10:11.343

0

That's not valid XML. Where does the <br> close? If it's <br/> then it will be empty because the following sibling is the <br/>.

edited Feb 06 '14 at 23:10

answered Feb 06 '14 at 22:54

helderdarocha

this is the html that i got from the webpage. i can't change it and even if i could, i don't want. i want to extract the data from the website, note fixing their html :) – Marco Dinatsoli Feb 06 '14 at 22:55
If the HTML is not well-formed XML it won't parse and you won't be able to use XPath. You'll have to use something else. Are you sure it's not a
instead of
? – helderdarocha Feb 06 '14 at 22:57
If it is not well-formed, and if you can't fix it, you can try to convert HTML into well-formed XHTML using [JTidy](http://jtidy.sourceforge.net/) first. After that you can use XPath. – helderdarocha Feb 06 '14 at 22:59
yes it is
and you should know that every problem has a solution, so please don't tell that `it won't parse` because it will be parsed :P – Marco Dinatsoli Feb 06 '14 at 23:00
lets wait for an answer that doesn't use `xhtml`. i have a lot of pages and my system will have performance problem if i covered my html for every item. – Marco Dinatsoli Feb 06 '14 at 23:01
Well, supposing something there closes the
for you to make it work with XPath, then you have to select the node after it: `//em[normalize-space(.) = "Description"]/following-sibling::node()[3]` since `node[2]` is the `
` – helderdarocha Feb 06 '14 at 23:08
I found the solution and I wrote it in an answer – Marco Dinatsoli Feb 06 '14 at 23:13

2 Answers2