Is there a way to simplify this? remove contents from text, python

Question

I've only started writing python for a week and I'm working on parsing and web scrape a website. How do I get rid of all the contents? tried a couple of things but I can't seem to figure it out. Help would be greatly appreciate!

import re
import requests
import json
from bs4 import BeautifulSoup, NavigableString

url = 'http://www.grabexample.com'
geturl = requests.get(url)
some_text = geturl.text
soup = BeautifulSoup(some_text, "html.parser")
soup.prettify()

all_on_URL = soup.find_all('a')
grab_text = soup.get_text(strip=True)

parse_a_text = grab_text.replace("</a>", '').replace("<a>", '').replace("<a", '')
parse_p_text = parse_a_text.replace("</p>", '').replace("<p>", '').replace("<p", '')
parse_div_text = parse_p_text.replace("</div>", '').replace("<div>", '').replace("<div", '')
parse_li_text = parse_div_text.replace("</li>", '').replace("<li>", '').replace("<li", '')
parse_span_text = parse_li_text.replace("</span>", '').replace("<span>", '').replace("<span", '')
parse_img_text = parse_span_text.replace("</img>", '').replace("<img>", '').replace("<img", '')
parse_ul_text = parse_img_text.replace("</ul>", '').replace("<ul>", '').replace("<ul", '')
parse_ol_text = parse_ul_text.replace("</ol>", '').replace("<ol>", '').replace("<ol", '')
parse_label_text = parse_ol_text.replace("</label>", '').replace("<label>", '').replace("<label", '')
parse_h_text = parse_ol_text.replace("</h>", '').replace("<h>", '').replace("<h", '')
parse_h1_text = parse_h_text.replace("</h1>", '').replace("<h1>", '').replace("<h1", '')
parse_h2_text = parse_h1_text.replace("</h2>", '').replace("<h2>", '').replace("<h2", '')
parse_h3_text = parse_h2_text.replace("</h3>", '').replace("<h3>", '').replace("<h3", '')
parse_h4_text = parse_h3_text.replace("</h4>", '').replace("<h4>", '').replace("<h4", '')
parse_h5_text = parse_h4_text.replace("</h5>", '').replace("<h5>", '').replace("<h5", '')
parse_href_text = parse_h4_text.replace("href=", '').replace("<", '').replace(">", '')
parse_box_text = parse_h4_text.replace("[]", '').replace("[", '').replace("]", '')
parse_space_text = parse_box_text.replace("\n", "").replace("   ", "")
parse_colon_text = parse_space_text.replace("{", '').replace("}", '').replace("#", '')

print(parse_colon_text)

Another way I tried to write it but didn't work, possibly I did something wrong here?

def notusetags():
        invalid_tags= ['a', 'div', 'span', 'class', 'p', 'img', 'li', '\n']
        # get_text(strip=True)
        for tag in invalid_tags:
            for match in soup2.findAll(tag):
                match.replaceWithChildren('')

        print soup

Some other ways I tried to write it but it didn't work, possibly I missing something here too?

invalid_tags= ['a', 'div', 'span', 'class', 'p', 'img', 'li', '\n']
# get_text(strip=True)
for stripped in parse_text:
    if stripped.name in invalid_tags:
        s=""
        for c in stripped.contents:
            if not isinstance(c, NavigableString):
                c = stripped(unicode(c), invalid_tags)
            s+= unicode(c)
        stripped.replaceWith(s)
        testtext.append(stripped)

testtext = []

print(testtext)

you're already loading BeautifulSoup, why aren't you using it to do the parsing? That's what it's for ... — Cfreak, Apr 04 '17 at 20:23
Here's a pretty good answer on the same subject: https://stackoverflow.com/questions/1765848/remove-a-tag-using-beautifulsoup-but-keep-its-contents — Cfreak, Apr 04 '17 at 20:28
I'm trying to do a generic search so that it can scrape and parse any websites — Sou Yang, Apr 04 '17 at 20:35

score 0 · Accepted Answer · answered Apr 06 '17 at 11:14

So your first code looks pretty ugly (and also contains two copy&paste errors as far as I can tell, look at the lines parse_href_text = ... and parse_box_text =). As others have mentioned in the comments, using a library for parsing the website might be the better choice.

But let's take a look at your code and try to simplify it:

so first you've got different tags, and you want to delete </tag>, <tag> and <tag for each of them. You can write this as a function:

def remove_tag(text, tag):
    return text.replace("</{}>".format(tag), '').replace("<{}>".format(tag), '').replace("<{}".format(tag), '')

then call this function for all of your tags:

tag_list = ['a', 'p', 'div', 'li', 'span', 'img', 'ul', 'ol', 'label', 'h', 'h1', 'h2', 'h3', 'h4', 'h5']

for tag in tag_list:
    text = remove_tag(text, tag)

and then, remove the other items:

others = ['href=', '<', '>', '[]', '[', ']', '\n', '   ', '{', '}', '#']

for s in others:
    text = text.replace(s, '')

Putting it all together:

import re
import requests
import json
from bs4 import BeautifulSoup, NavigableString

url = 'http://www.grabexample.com'
geturl = requests.get(url)
some_text = geturl.text
soup = BeautifulSoup(some_text, "html.parser")
soup.prettify()

all_on_URL = soup.find_all('a')
grab_text = soup.get_text(strip=True)

def remove_tag(text, tag):
    return text.replace("</{}>".format(tag), '').replace("<{}>".format(tag), '').replace("<{}".format(tag), '')

tag_list = ['a', 'p', 'div', 'li', 'span', 'img', 'ul', 'ol', 
            'label', 'h', 'h1', 'h2', 'h3', 'h4', 'h5']
for tag in tag_list:
    grab_text = remove_tag(grab_text, tag)

others = ['href=', '<', '>', '[]', '[', ']', '\n', '   ', '{', '}', '#']
for s in others:
    grab_text = grab_text.replace(s, '')

print(grab_text)

Is there a way to simplify this? remove contents from text, python

1 Answers1