Non-greedy XPATH to get HTML before the nearest h2 node

Question

Is it possible to scrape XPATH non-greedy-ly? I mean for example I have this HTML:

<div>
    <p>A</p>
    <p>B</p>
    <h2>Only until this node</h2>
    <p>I should not get this</p>
    <h2>Even though this node exists</h2>
</div>

I want an XPATH which only gets the paragraphs with A and B inside. The text inside the nearest h2 node is always changing, so I need non-greedy XPATH if it is possible. Is it possible? And how?

score 2 · Accepted Answer · answered Dec 16 '18 at 00:12

I assume <h2>Only until this node</h2> is dynamic, you can select first index of h2

//div/h2[1]/preceding-sibling::p

var htmlString = `
<body>
  <div>
    <p>A</p>
    <p>B</p>
    <h2>Only until this node</h2>
    <p>I should not get this</p>
    <h2>Even though this node exists</h2>
  </div>
  <div>
    <p>A1</p>
    <p>B2</p>
    <p>C3</p>
    <h2>Second Only until this node</h2>
    <p>I should not get this</p>
    <h2>Even though this node exists</h2>
  </div>
</body>`;

var doc = new DOMParser().parseFromString(htmlString, 'text/xml');
var iterator = doc.evaluate('//div/h2[1]/preceding-sibling::p', doc, null, XPathResult.UNORDERED_NODE_ITERATOR_TYPE, null);
var thisNode = iterator.iterateNext();
while (thisNode) {
  console.log(thisNode.outerHTML);
  thisNode = iterator.iterateNext();
}

Cool! You could guess what I was thinking! Yeah, the second `h2` node is dynamic. — Aminah Nuraini, Dec 18 '18 at 08:45

score 1 · Answer 2 · answered Dec 15 '18 at 07:22

1

Try this xpath

//div/p[following::h2[contains(.,'Only until this node')]]

to get desired content out of the html elements until it hits the p element containing this text Only until this node.

Check out the example below:

from scrapy import Selector

htmldoc="""
<div>
    <p>A</p>
    <p>B</p>
    <p>C</p>
    <p>D</p>
    <h2>Only until this node</h2>
    <p>E</p>
    <p>F</p>
    <p>I should not get this</p>
    <h2>Even though this node exists</h2>
    <p>I should not even this</p>
</div>
"""
sel = Selector(text=htmldoc)
for item in sel.xpath("//div/p[following::h2[contains(.,'Only until this node')]]/text()").extract():
    print(item)

What it produces:

A
B
C
D

answered Dec 15 '18 at 07:22

SIM

21,997
5
37
109

Thanks a lot. I have installed Selector and I got 'ModuleNotFoundError: No module named 'scrapy'' – YasserKhalil Dec 16 '18 at 18:44
Is this related to specific version of python. I am using 3.7 – YasserKhalil Dec 17 '18 at 03:55
No. Any version will do. Perhaps you need to have `twisted` installed first. – SIM Dec 17 '18 at 07:25
I am using pycharm and I got errors when trying to install 'Twisted' package too .. May be the problem with the IDE? .. What is the best IDE? – YasserKhalil Dec 17 '18 at 15:07
Are you not using cmd to perform those installations? – SIM Dec 17 '18 at 15:22
I tried both pycharm and cmd ... and the same result of errors at the end of both – YasserKhalil Dec 17 '18 at 15:31
I have found solution at this [LINK](https://stackoverflow.com/questions/29846087/microsoft-visual-c-14-0-is-required-unable-to-find-vcvarsall-bat) In Lalit Kumar B's comment – YasserKhalil Dec 17 '18 at 19:24

score 0 · Answer 3 · answered Dec 15 '18 at 00:49

0

You can try the following XPath-1.0 expression:

/div/p[following-sibling::*[self::h2='Only until this node']]

It gets all p elements which have a h2 successor with the text() value "Only until this node".

answered Dec 15 '18 at 00:49

zx485

28,498
28
50
59

Non-greedy XPATH to get HTML before the nearest h2 node

3 Answers3