84

I would like to scrape a list of items from a website, and preserve the order that they are presented in. These items are organized in a table, but they can be one of two different classes (in random order).

Is there any way to provide multiple classes and have BeautifulSoup4 find all items which are in any of the given classes?

I need to achieve what this code does, except preserve the order of items as it was in the source code:

items = soup.findAll(True,{'class':'class1'})
items += soup.findAll(True,{'class':'class2'})
alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
sebo
  • 1,584
  • 4
  • 16
  • 19
  • Thank you alecxe and Roman Pekar for helping me solve this. I was able to achieve what I wanted using partial class names and the additional check as suggested in alecxe's updated answer. – sebo Sep 10 '13 at 19:01

6 Answers6

116

you can do this

soup.findAll(True, {'class':['class1', 'class2']})

example:

>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup('<html><body><div class="class1"></div><div class="class2"></div><div class="class3"></div></body></html>')
>>> soup.findAll(True, {"class":["class1", "class2"]})
[<div class="class1"></div>, <div class="class2"></div>]
Roman Pekar
  • 107,110
  • 28
  • 195
  • 197
  • In my code this returns an empty list the same way as alecxe's solution. See comment under his response for my code. – sebo Sep 10 '13 at 18:15
  • 2
    @sebo try this: `soup.findAll(True, {"class":["equal", "up"]})`. – alecxe Sep 10 '13 at 18:30
  • @alecxe this works, +1 for you, but still trying to find how to make this work with full class names – Roman Pekar Sep 10 '13 at 18:34
  • 1
    @RomanPekar thank you. I think `bs4` doesn't apply `class` filter to the whole `class` attribute value, it splits classes by space. See http://stackoverflow.com/questions/1242755/beautiful-soup-cannot-find-a-css-class-if-the-object-has-other-classes-too. – alecxe Sep 10 '13 at 18:51
  • @RomanPekar I think there is one way to workaround it - add an additional check inside the loop, see my updated answer. What do you think? – alecxe Sep 10 '13 at 18:56
  • @alecxe well, this would work I think, but I'd rather have more pythonic solution. HAve to to some work now, try to search later – Roman Pekar Sep 10 '13 at 18:59
28

I am new to Python with BeautifulSoup but may be my answer help you. I came across the same situation where I have to find multiple classes of one tag so, I just pass the classes into an array and it works for me. Here is the code snippet

# Search with single Class
    find_all("tr",  {"class":"abc"})
# Search with multiple classes
    find_all("tr",  {"class": ["abc", "xyz"]})
joshpetit
  • 677
  • 6
  • 17
Bhoopi
  • 6,523
  • 3
  • 22
  • 16
17
    <html>
        <body>
            <div class="cls1">ok</div>
            <div class="cls2">hi</div>
            <div class="cls1 cls2">both</div>
        </body>
    </html>

OR operator

    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html)
    divs = soup.find_all('div', class_=['cls1', 'cls2'])
    print(divs)

output:

[<div class="cls1">ok</div>, <div class="cls2">hi</div>, <div class="cls1 cls2">both</div>]

AND operator

    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html)
    divs = soup.select('div.cls1.cls2')
    print(divs)

output:

[<div class="cls1 cls2">both</div>]
danilo
  • 7,680
  • 7
  • 43
  • 46
15

Or this with the more recent version of BeautifulSoup:

find_all('a', class_=['class1', 'class2'])

Using "class" would return an error so they use "class_" instead.

Abdelghani Bekka
  • 576
  • 9
  • 18
14

One way to do it is to use regular expression instead of a class name:

import re
import requests
from bs4 import BeautifulSoup


s = requests.Session()
link = 'https://leaderboards.guildwars2.com/en/na/achievements'
r = s.get(link)


soup = BeautifulSoup(r.text)
for item in soup.findAll(True, {"class": re.compile("^(equal|up)$")}):
    if 'achievements' in item.attrs['class'] and 'number' in item.attrs['class']:
        print item
alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
  • Thank you for the quick response. Right now, the call does not return anything though. Is it possible that this is caused by there being spaces in the class names? (ex. "class 1") Sorry, I know nothing about regular expressions. – sebo Sep 10 '13 at 18:04
  • @sebo could you show the code you are using so I can reproduce and fix the problem? – alecxe Sep 10 '13 at 18:07
  • When stripped down, this is the code I'm looking at: `import requests from bs4 import BeautifulSoup s = requests.Session() link = 'https://leaderboards.guildwars2.com/en/na/achievements' r = s.get(link) soup = BeautifulSoup(r.text) items = soup.findAll(True, {"class":["equal achievements number", "up achievements number"]}) ` This one is applying Roman Pekar's solution, but it returns an empty list. The same thing happens using the regular expression solution. Retrieving either class one at a time works fine though. Sorry for the terrible formatting. – sebo Sep 10 '13 at 18:17
  • 1
    @sebo is it working `soup.findAll(True, {"class": ".*achievements number.*"})`? – alecxe Sep 10 '13 at 18:20
  • That also returns an empty list. – sebo Sep 10 '13 at 18:23
  • This updated answer works for finding all the fields which contain "equal" or "up", but there are some fields on the page that I am not interested in which have that. I can probably go from here by using list slicing, but I'm still curious if there is a way to provide full class names. – sebo Sep 10 '13 at 18:46
  • @sebo please see updated answer. It's a workaround that should work. – alecxe Sep 10 '13 at 18:50
  • This answer is not what I was looking for at all. But it helped me realize that I can simply specify a space-separated list of classes. Just what I needed. Because soup.select, for example, doesn't seem to be able to work with a space-separated list of CSS selectors – MickeyDickey Dec 12 '21 at 22:16
2

If you're working with an Url as parameter dont forget to pass the headers too. I was fighting for like one hour to get these div elements with 2 classes and it wasnt working for mi till i noticed that i forget to pass the this headers.

header = {
    "Accept-Language": "es-ES,es;q=0.9",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36"
}
url = 'something.com'
response = requests.get(url=url,headers=header)
response.raise_for_status()
data = response.text

soup = BeautifulSoup(data, 'html.parser')  

elements = soup.select('div.fde444d7ef._c445487e2')
Jose Francisco Cheble
  • 67
  • 4