1

I'm trying to find all images (.png, .bmp, .jpg) and executables (.exe) from anchor links using lxml. From this similar thread, the accepted answer suggests doing something like this:

png = tree.xpath("//div/ul/li//a[ends-with(@href, '.png')]")
bmp = tree.xpath("//div/ul/li//a[ends-with(@href, '.bmp')]")
jpg = tree.xpath("//div/ul/li//a[ends-with(@href, '.jpg')]")
exe = tree.xpath("//div/ul/li//a[ends-with(@href, '.exe')]")

However, I get keep getting this error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lxml.etree.pyx", line 2095, in lxml.etree._ElementTree.xpath (src/lxml/lxml.etree.c:53597)
  File "xpath.pxi", line 373, in lxml.etree.XPathDocumentEvaluator.__call__ (src/lxml/lxml.etree.c:134052)
  File "xpath.pxi", line 241, in lxml.etree._XPathEvaluatorBase._handle_result (src/lxml/lxml.etree.c:132625)
  File "xpath.pxi", line 226, in lxml.etree._XPathEvaluatorBase._raise_eval_error (src/lxml/lxml.etree.c:132453)
lxml.etree.XPathEvalError: Unregistered function

I'm running lxml 3.2.4 through pip.

Also, instead of defining the xpath 4 times for each file extension, is there a way to use xpath and specify all four file extensions at once?

Community
  • 1
  • 1
user21398
  • 1,473
  • 4
  • 18
  • 31

2 Answers2

3

ends-with is a function defined for XPath 2.0, XQuery 1.0 and XSLT 2.0 while lxml only supports XPath 1.0, XSLT 1.0 and the EXSLT extensions. So you couldn't use this function. The doc is here and here.

You could use regular expressions in XPATH. The following is a sample code which returns nodes matching the regular expressions: 

regexpNS = 'http://exslt.org/regular-expressions'
tree.xpath("//a[re:test(@href, '(png|bmp|jpg|exe)$')]", namespaces={'re':regexpNS}")

Here is a similar question Python, XPath: Find all links to images and regular-expressions-in-xpath

Community
  • 1
  • 1
flyer
  • 9,280
  • 11
  • 46
  • 62
0

I think that is a problem with an external library not recognizing the ends-with function. The documentation discusses working with links. I think a nicer solution would be something like:

from urlparse import urlparse
tree.make_links_absolute(base_href='http://example.com/')
links = []
for i in tree.iterlinks():
    url = urlparse(i[2])  # ensures you are getting the remote file path
    if url.path.endswith('.png') or url.path.endswith('.exe') ... :
        # there are other ways you could filter the links here
        links.append(i[2])
ChrisP
  • 5,812
  • 1
  • 33
  • 36
  • If i know where the links live (ie, inside an unordered list `"//div/ul/li//a"`), is there a way to get `iterlinks()` to search only through the unordered lists instead of the entire dom? – user21398 Dec 11 '13 at 21:02
  • Hm.. when I do `tree.iterlinks()`, I get this error: `AttributeError: 'lxml.etree._ElementTree' object has no attribute 'iterlinks'.` – user21398 Dec 11 '13 at 21:40