Python BeautifulSoup Parsing Not working

Question

I have mainly used python for data anlaysis and new to scraping. I am trying to learn the BeautifulSoup package. I am having problems to make the following code work.

from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen('http://pythonscraping.com/pages/warandpeace.html')
bsobj = BeautifulSoup(html)
name_list = bsobj.findAll('span',{'class':'green'})

I am getting an empty list.

It's clear that the problem comes from the 4th line. I am not sure why. Everything is standard here. I don't know what went wrong.

bsobj.prettify()

Returns ''

But when I do html.read(), I can see all the html code fine. The problem is not solved by the answers below. The problem clearly comes from line4. It doesn't matter if I use bsobj.findAll(), or bsobj.find_all(). They are equilvalent and as I mentioned, the bsobj.prettify() returns ''.

add `print(bsobj.prettify())` and post output – MD. Khairul Basar Sep 10 '17 at 07:01 — MD. Khairul Basar, Sep 10 '17 at 07:01
If `bsobj` is `None` print `html`. – MD. Khairul Basar Sep 10 '17 at 07:02 — MD. Khairul Basar, Sep 10 '17 at 07:02
`bsobj = BeautifulSoup(html.read(), 'html.parser')` – Burhan Khalid Sep 10 '17 at 07:15 — Burhan Khalid, Sep 10 '17 at 07:15

Sam Chats · Answer 1 · 2017-09-10T07:09:17.377

0

I think that line should be bsobj = BeautifulSoup(html.read())

edited Sep 10 '17 at 07:09

answered Sep 10 '17 at 07:03

Sam Chats

2,271
1
12
34

'html.parser' is actually not a good parser. `'lxml'` is better, and it is the default choice, anyway. – DYZ Sep 10 '17 at 07:05
@DYZ I was about to add it! :) I myself use more `lxml` - see my GitHub :) – Sam Chats Sep 10 '17 at 07:06
@YanSong is your problem fixed? – Sam Chats Sep 10 '17 at 07:12

score 0 · Answer 2 · answered Sep 10 '17 at 07:05

The findall is wrong...

bsobj.find_all('span',{'class':'green'})

It returns

[<span class="green">Anna
 Pavlovna Scherer</span>, <span class="green">Empress Marya
 Fedorovna</span>, <span class="green">Prince Vasili Kuragin</span>, <span class="green">Anna Pavlovna</span>, <span class="green">St. Petersburg</span>, <span class="green">the prince</span>, <span class="green">Anna Pavlovna</span>, <span class="green">Anna Pavlovna</span>, <span class="green">the prince</span>, <span class="green">the prince</span>, <span class="green">the prince</span>, <span class="green">Prince Vasili</span>, <span class="green">Anna Pavlovna</span>, <span class="green">Anna Pavlovna</span>, <span class="green">the prince</span>, <span class="green">Wintzingerode</span>, <span class="green">King of Prussia</span>, <span class="green">le Vicomte de Mortemart</span>, <span class="green">Montmorencys</span>, <span class="green">Rohans</span>, <span class="green">Abbe Morio</span>, <span class="green">the Emperor</span>, <span class="green">the prince</span>, <span class="green">Prince Vasili</span>, <span class="green">Dowager Empress Marya Fedorovna</span>, <span class="green">the baron</span>, <span class="green">Anna Pavlovna</span>, <span class="green">the Empress</span>, <span class="green">the Empress</span>, <span class="green">Anna Pavlovna's</span>, <span class="green">Her Majesty</span>, <span class="green">Baron
 Funke</span>, <span class="green">The prince</span>, <span class="green">Anna
 Pavlovna</span>, <span class="green">the Empress</span>, <span class="green">The prince</span>, <span class="green">Anatole</span>, <span class="green">the prince</span>, <span class="green">The prince</span>, <span class="green">Anna
 Pavlovna</span>, <span class="green">Anna Pavlovna</span>]

Python BeautifulSoup Parsing Not working

2 Answers2