Disable special "class" attribute handling

Question

The Story:

When you parse HTML with BeautifulSoup, class attribute is considered a multi-valued attribute and is handled in a special manner:

Remember that a single tag can have multiple values for its “class” attribute. When you search for a tag that matches a certain CSS class, you’re matching against any of its CSS classes.

Also, a quote from a built-in HTMLTreeBuilder used by BeautifulSoup as a base for other tree builder classes, like, for instance, HTMLParserTreeBuilder:

# The HTML standard defines these attributes as containing a
# space-separated list of values, not a single value. That is,
# class="foo bar" means that the 'class' attribute has two values,
# 'foo' and 'bar', not the single value 'foo bar'.  When we
# encounter one of these attributes, we will parse its value into
# a list of values if possible. Upon output, the list will be
# converted back into a string.

The Question:

How can I configure BeautifulSoup to handle class as a usual single-valued attribute? In other words, I don't want it to handle class specially and consider it a regular attribute.

FYI, here is one of the use-cases when it can be helpful:

BeautifulSoup returns empty list when searching by compound class names

What I've tried:

I've actually made it work by making a custom tree builder class and removing class from the list of specially-handled attributes:

from bs4.builder._htmlparser import HTMLParserTreeBuilder

class MyBuilder(HTMLParserTreeBuilder):
    def __init__(self):
        super(MyBuilder, self).__init__()

        # BeautifulSoup, please don't treat "class" specially
        self.cdata_list_attributes["*"].remove("class")


soup = BeautifulSoup(data, "html.parser", builder=MyBuilder())

What I don't like in this approach is that it is quite "unnatural" and "magical" involving importing "private" internal _htmlparser. I hope there is a simpler way.

^{NOTE: I want to save all other HTML parsing related features, meaning I don't want to parse HTML with "xml"-only features (which could've been another workaround).}

I thought it was a bug when I saw your avatar under a beautifulsoup question with no answer and then I realized you *asked* the question! I can't help you everything I tried didn't work or involved two iterations. — dstudeba, Dec 15 '15 at 18:06
I do not know how to do this, but for the specific use case provided as example I provided a different answer (so I posted it there). It is simpler in my opinion but may not be sufficient for other use cases — rll, Dec 18 '15 at 10:51
Using it as a css selector?. Maybe in that case the simplest option could be not to use a common class selector, but an attribute selector. Selector '.myclass' is just the same that '[class=~"myclass"]', but selector '[class="class"]' is an element whose "class" attribute value is exactly equal to "myclass" (not myclass in a space sepated list). — miguel-svq, Dec 19 '15 at 18:59
@miguel-svq good point, CSS selectors might help here depending on the use case. If we take that use case from a linked question, there is a regular expression pattern applied to a class attribute - this is something we cannot really achieve with CSS selectors, but, we can use them to narrow down the search and then do some manual checks. Thank you! — alecxe, Dec 19 '15 at 19:26

score 6 · Accepted Answer · edited Aug 30 '16 at 18:47

What I don't like in this approach is that it is quite "unnatural" and "magical" involving importing "private" internal _htmlparser. I hope there is a simpler way.

Yes, you can import it from bs4.builder instead:

from bs4 import BeautifulSoup
from bs4.builder import HTMLParserTreeBuilder

class MyBuilder(HTMLParserTreeBuilder):
    def __init__(self):
        super(MyBuilder, self).__init__()
        # BeautifulSoup, please don't treat "class" as a list
        self.cdata_list_attributes["*"].remove("class")


soup = BeautifulSoup(data, "html.parser", builder=MyBuilder())

And if it's important enough that you don't want to repeat yourself, put the builder in its own module, and register it with register_treebuilders_from() so that it takes precedence.

While this works, I hate that my IDEs (PyCharm and IntelliJ IDEA with Python plugin) complain about the `bs4.builder` import. They say "Unresolved reference 'HTMLParserTreeBuilder'" and it can't jump to the declaration for it when I ask for it. Are other IDEs any better about this? — Mr. Lance E Sloan, Aug 30 '16 at 18:13

score 2 · Answer 2 · answered Dec 18 '15 at 16:07

The class HTMLParserTreeBuilder is actually declared on the upper module _init__.py, so there is no need to import directly from the private submodule. That said I would do it the following way:

import re

from bs4 import BeautifulSoup
from bs4.builder import HTMLParserTreeBuilder

bb = HTMLParserTreeBuilder()
bb.cdata_list_attributes["*"].remove("class")

soup = BeautifulSoup(bs, "html.parser", builder=bb)
found_elements = soup.find_all(class_=re.compile(r"^name\-single name\d+$"))
print found_elements

It is basically the same as defining the class as in the OP (maybe a bit more explicit), but I don't think there is a better way to do it.

Disable special "class" attribute handling

2 Answers2

Linked