Removing punctuation from a list in python

Question

I know this is a common question but I haven't found an applicable answer. I'm trying to remove the punctuation from a list of words, which I have gotten from scraping an HTML page in an earlier function. Here is what I have:

import re
def strip_text():    
        list_words = get_text().split()
        print(list_words)
        for i in range(len(list_words)):
            list_words = re.sub("[^a-zA-Z]"," ",list_words)
            list_words = list_words.lower()
        return list_words
    print(get_text()) 
    print(strip_text())

I realize that this doesn't work because the re.sub bit is supposed to be used on a string, not a list. Is there an equally efficient way to do this? Should I make the list of words a string again?

edit: this problem is scraping the text from an HTML page, like I said. The code before what I have above looks like this:

from bs4 import BeautifulSoup
import requests
from collections import Counter
import re
tokens = []
types= Counter(tokens)
#str_book = ""
str_lines = ""
import string

def get_text(): 
   # str_lines = ""
    url = 'http://www.gutenberg.org/files/1155/1155-h/1155-h.htm'
    r = requests.get(url)
    data = r.text
    soup = BeautifulSoup(data, 'html.parser')
    text = soup.find_all('p') #finds all of the text between <p>
    i=0
    for p in text:
        i+=1
        line = p.get_text()
        if (i<10):
            continue
        print(line)
    return line

So the list of words would be a list of all the words in the Agatha Christie book that I'm using. Hopefully that helps.

Answer 1

You don't need regex at all. string.punctuation contains all of the punctations. Just iterate and skip those.

>>> import string
>>> ["".join( j for j in i if j not in string.punctuation) for i in  lst]

Answer 2

Taking a look at get_text(), it appears we need to modify a few things before we can remove any punctuation. I've added some comments in here.

def get_text(): 
    str_lines = []  # create an empty list
    url = 'http://www.gutenberg.org/files/1155/1155-h/1155-h.htm'
    r = requests.get(url)
    data = r.text
    soup = BeautifulSoup(data, 'html.parser')
    text = soup.find_all('p') #finds all of the text between <p>
    i=0
    for p in text:
        i+=1
        line = p.get_text()
        if (i<10):
            continue
        str_lines.append(line)  # append the current line to the list
    return str_lines  # return the list of lines

First, I uncommented your str_lines variable and set it to an empty list. Next, I replaced the print statement with code to append the line to the list of lines. Finally, I changed the return statement to return that list of lines.

For strip_text(), we can reduce it to a few lines of code:

def strip_text():    
    list_words = get_text()
    list_words = [re.sub("[^a-zA-Z]", " ", s.lower()) for s in list_words]
    return list_words

There is no need to operate on a per-word basis because we can look at the entire line and remove all punctuation, so I removed the split(). Using list comprehension, we can alter every element of the list in a single line, and I also put the lower() method in there to condense the code.

To implement the answer provided by @AhsanulHaque, you just need to substitute that second line of the strip_text() method with it, as shown:

def strip_text():
    list_words = get_text()
    list_words = ["".join(j.lower() for j in i if j not in string.punctuation)
                  for i in list_words]
    return list_words

---

For fun, here is that translate method I mentioned earlier implemented for Python 3.x, as described here:

def strip_text():
    list_words = get_text()
    translator = str.maketrans({key: None for key in string.punctuation})
    list_words = [s.lower().translate(translator) for s in list_words]
    return list_words

Unfortunately I cannot time any of these for your particular code because Gutenberg blocked me temporarily (too many runs of the code too quickly, I suppose).