Python to remove all
instances in a string using beautifulsoup

Question

I am using the code below to get the following

<p>Ibn Umar reported: I passed by the Messenger of Allah, peace and blessings be
 upon him, while my garment was trailing. The Prophet said, ÔÇ£<b>O Abdullah, ra
ise your garment</b>.ÔÇØ I lifted it up and he told me to raise it higher and I
did so. Some of the people said, ÔÇ£To where should it be raised?ÔÇØ The Prophet
 said, ÔÇ£<b>In the middle of the shins</b>.ÔÇØ</p>

I am wondering if you would be able to help me get rid of the <p>, </p> and <b>

Code:

url1 = "http://www.dailyhadithonline.com/2013/07/21/hadith-on-clothing-the-lower-garment-should-be-hallway-between-the-shins/"
content1 = urllib2.urlopen(url1).read()
soup = BeautifulSoup(content1) 
english_hadith = soup.findAll('p')[0]
print english_hadith

James Mills · Answer 1 · 2013-12-10T02:33:02.920

1

You can do this with nltk.

Example:

from nltk import clean_html
html = "..."
clean_html(html)

edited Dec 10 '13 at 02:33

answered Dec 10 '13 at 02:31

James Mills

18,669
3
49
62

nltk is a best library to remove html tags. – Nilesh Dec 10 '13 at 03:39
But Avoid package external python library, If you can use python module like re for this. If there is no option in python, then only you can use external python library – Nilesh Dec 10 '13 at 03:48
Sure you could write some hand-written code, or borrow someone else's but there are nicely libraries for this such as nltk that just do the job for you :) (*why re-invent the wheel?*) – James Mills Dec 10 '13 at 03:59

rjv · Accepted Answer · 2013-12-10T02:56:07.223

0

I would recommend using regular expressions rather than beautifulsoup.

>>> import re
>>> a='<p>dhhdhd<p>dhdhd</p>'
>>> re.sub('<p>|</p>','',a)
'dhhdhddhdhd'

A more general regular expression would be

re.sub('<p[^>]*>|</p>','',a)

edited Dec 10 '13 at 02:56

answered Dec 10 '13 at 02:33

rjv

6,058
5
27
49

Why? regex sucks for parsing html. What if there are other possibly nested tags? – John La Rooy Dec 10 '13 at 02:37
i believe the OP needed to eliminate just `p` and `b` tags,as per the text example given – rjv Dec 10 '13 at 02:38
@rjv: A larger issue would be things like: `
` and `
`. But that doesn't appear to be a problem for Ossama.
– Bill Lynch Dec 10 '13 at 02:39
you will have to modify the regexp in that case – rjv Dec 10 '13 at 02:47
Which is another reason to just use BeautifulSoup in the first place – John La Rooy Dec 10 '13 at 02:49
i have modified the regexp. – rjv Dec 10 '13 at 02:57

score 0 · Answer 3 · answered Dec 10 '13 at 02:35

0

You were close.

print english_hadith.text

Displays:

Ibn Umar reported: I passed by the Messenger of Allah, peace and blessings be upon him, while my garment was trailing. The Prophet said, “O Abdullah, raise your garment.” I lifted it up and he told me to raise it higher and I did so. Some of the people said, “To where should it be raised?” The Prophet said, “In the middle of the shins.”

answered Dec 10 '13 at 02:35

John La Rooy

295,403
53
369
502

I get the following UnicodeEncodeError: 'charmap' codec can't encode character u'\u201c' in position 137: character maps to – Ossama Dec 10 '13 at 02:36
@Ossama, what is your `sys.stdout.encoding`? – John La Rooy Dec 10 '13 at 02:38
@Ossama, `import sys; print sys.stdout.encoding` – John La Rooy Dec 10 '13 at 02:41
the sys.stdout.encoding is cp850 – Ossama Dec 10 '13 at 02:43
@Ossama, Your problem is that [your terminal](http://en.wikipedia.org/wiki/Code_page_850) doesn't support the fancy quote characters. – John La Rooy Dec 10 '13 at 02:47
Thanks.I'll try at me home computer latter. thanks for you help – Ossama Dec 10 '13 at 02:47
@Ossama, [This question](http://stackoverflow.com/questions/1650369/php-utf-8-to-msdos-command-line-encoding) provides a possible workaround – John La Rooy Dec 10 '13 at 02:48

Python to remove all instances in a string using beautifulsoup

3 Answers3

Python to remove all
instances in a string using beautifulsoup