11
>>> from lxml.etree import HTML, tostring
>>> tostring(HTML('<fb:like>'))
'<html><body><like/></body></html>'

Note how the tag turns from <fb:like> to simply <like>.

This makes processing pages that incorporate XFBML with lxml much harder. (Same thing happens to <g:plusone></g:plusone>)

Any help is appreciated.

Sergey Schetinin
  • 111
  • 6
  • 2
    HTML doesn't have namespaces. Welcome to the consequences of expando-elements. If it is XHTML + FBML (instead of HTML + FBML) then you could use an XML parser. – Quentin Jul 06 '11 at 15:21
  • I would have, if it was XML. The point of resilient HTML parsers is that they get the job done. While I don't like these magic tag names either, I think it's reasonable to expect from lxml to preserve them. I'll try to talk to the maintainers to see if this can be accommodated. – Sergey Schetinin Jul 06 '11 at 16:09

2 Answers2

2

One way to fix this issue is to patch libxml2.

Referring to the source code of libxml2.9.2 (https: //git.gnome.org/browse/libxml2/tree/?id=v2.9.2), in SAX2.c (https: //git.gnome.org/browse/libxml2/tree/SAX2.c?id=v2.9.2) (the internal SAX parser used to create the DOM tree) at line 1699 attributes with xmlns are not parsed when in HTML mode, and they are parsed like any other attributes at line and 1740. Consequently, it makes sense to adjust line 1622, which splits the name into prefix and local part. Change:

name = xmlSplitQName(ctxt, fullname, &prefix);

into

if (!ctxt->html) {
    name = xmlSplitQName(ctxt, fullname, &prefix);
} else {
    name = xmlStrdup(fullname);
    prefix = NULL;
}

Then libxml2 will consider tags such as <o:p> to be for elements with name o:p, that is, the colon is included in the element name with no special meaning. This is the correct interpretation in HTML. For example, the HTML5 specification says:

In the HTML syntax, namespace prefixes and namespace declarations do not have the same effect as in XML. For instance, the colon has no special meaning in HTML element names.

Hopefully this change will be approved for a future version of libxml2. There is an open bug report (https: //bugzilla.gnome.org/show_bug.cgi?id=654146).

Insightfuls
  • 21
  • 4
  • While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. – Marcus Apr 14 '15 at 06:41
  • @Marcus OK, have added details here. Sorry about the ugly links. I don't have sufficient reputation to include them as real links. – Insightfuls Apr 15 '15 at 05:51
1

Try adding the namespace prefix definitions that are missing. lxml will avoid the namespaces otherwise, supposedly to make it easier for you.

Most likely the sites you try to parse will not contain these namespace definitions, so you should add them.

Something like this: xmlns:adlcp="http://xxx/yy/zzz"

Community
  • 1
  • 1
jmora
  • 491
  • 3
  • 14