check if string contains word in any variation

Question

I'm trying to build a bad words filter in python (I accept non-coded answers, just need to know more about an algorithm that would work) and I need to know how I can check if a string, contains a specific word in any variation.

For example, let's say my bad word array is: ['others','hello','banana']

And the String I need to check is: Thinking alike or understanding something in a similar way with others.

For now, I'm looping on the string by checking every time if any element of the array exists in the phrase, but what if I want to check variations of the array? Like 0th3rs,Oth3r5 for the first element? For now, I'm manually checking it by doing multiple if statements and replacing a with @ etc... But this would not be good for a production code since I cannot prevent every scenario of character replacing, So I thought of something like an array of objects, where the index is the letter, like A which contains an array of its variations and check it dynamically in the string, but would this take too much time? Since it needs to check every type of word variation? Or is this achievable and usable in a real scenario?

A good start would be to split the text into words and normalize the words. — Klaus D., Mar 27 '21 at 12:13
Except the proposals to make a replacement dict - you can go more extreme and do ocr eg. https://stackoverflow.com/a/22879053/11610186 — Grzegorz Skibinski, Mar 27 '21 at 12:22
To tokenize the text into words you can use [nltk](https://www.nltk.org/) — Paweł Rubin, Mar 27 '21 at 12:28
Be careful you don't fall foul of the [Scunthorpe problem](https://en.wikipedia.org/wiki/Scunthorpe_problem). — rossum, Mar 27 '21 at 13:53

score 1 · Answer 1 · answered Mar 27 '21 at 12:16

1

Have you try using replace()?

For example:

input="0th3rs"
replace_pair={'0':'o','3':'e'}
for old, new in replace_pair.items():
    input = input.replace(old, new)
print(input)

will give you "others"

You have to still provide the replacement pairs but that would be better than "if" statement.

answered Mar 27 '21 at 12:16

Cheukting

264
1
6

You're overloading python `input` function - you can do eg. `input_` instead – Grzegorz Skibinski Mar 27 '21 at 12:19
Thanks! Usually, my linter will pick that up for me. I am bad at naming – Cheukting Mar 28 '21 at 12:06

score 0 · Answer 2 · answered Mar 27 '21 at 12:10

0

can't you just extend your list of bad words to contain different variations?

bad_words = ["others", "0th3rs", "banana"]
text = "this is the text about bananas and 0th3rs"
for word in bad_words:
    if word in text:
        text = text.replace(word, "*flowers*")

answered Mar 27 '21 at 12:10

cold2x

1

You need to consider substrings. What if `text` is `"This bothers me"`? – snakecharmerb Mar 27 '21 at 12:19

Paweł Rubin · Accepted Answer · 2021-03-27T12:28:00.227

I cannot prevent every scenario of character replacing

That's true. However, you can handle the majority of scenarios.

I would consider declaring a mapping of replacements and their meaning:

REPLACEMENTS_DICT = {
    "@": "a",
    "4": "a",
    "3": "e",
    "0": "o",
    ...
}

Then, before checking if a particular string is inside the bad_word_array, one should translate the string with regard to the replacement dict and then make a case-insensitive comparison:

def translate(word: str) -> str:
    return "".join(REPLACEMENTS_DICT.get(c, c) for c in word).lower()

def is_bad_word(word: str) -> bool:
    return translate(word) in BAD_WORDS

Example

BAD_WORDS = ["others", "hello", "banana"]
print(is_bad_word("0th3rs"))  # True
print(is_bad_word("Oth3rs"))  # True

For tokenizing the text into words you can use nltk.

import nltk

sentence = "Thinking alike or understanding something in a similar way with others."

words = nltk.word_tokenize(sentence)

for word in words:
    assert is_bad_word(word)

check if string contains word in any variation

3 Answers3