How to extract meaningful words from messy strings?

Question

consider strings like the following, how can I extract any meaningful words?

ctl00_PD_lblProductTitle ---> 'Product Title' (ct100_PD_lbl is not a complete word, ignore)

prod-detail-info ---> 'Detail Info' (prod is not a complete word, ignore)

prodprice price-onsale ---> 'price on sale' (prod is not a complete word, ignore)

rating-score ---> 'Rating Score'

RatingScore ---> 'Rating Score' 

curious to know what technique or process this is called and any libraries if any. could regex be robust enough?

Answer 1

In short, the examples you provide cannot be assessed in the way you want them to be unless you skew your decision tree or classifier to your particular data. (Consider, for example, numbers 2 and 3. Prod is a word in any English dictionary, but info won't necessarily show up in most English dictionaries.)

If you train your classifier or decision tree on these specific data, only then will you get the result you want. But generally speaking, you might try tokenizing your text to begin with (as @user2314737 has suggested):

>>> import nltk
>>> t = '''ctl00_PD_lblProductTitle

prod-detail-info

prodprice price-onsale

rating-score

RatingScore'''
>>> nltk.tokenize.wordpunct_tokenize(t)
['ctl00_PD_lblProductTitle', 'prod', '-', 'detail', '-', 'info', 'prodprice', 'price', '-', 'onsale', 'rating', '-', 'score', 'RatingScore']

Then you might be able to find further possibile words with regular expressions such as this:

>>> re.findall(r'[A-Z][a-z]{2,}', 'ctl00_PD_lblProductTitle')  # also works for 'RatingScore'
['Product', 'Title']

This regular expression will find all sequences that begin with an upper-case letter and that are then followed by 2 or more lower-case letters. Regarding your comment, no, this regex will not work for unicode. Unfortunately, regular expressions in Python do not presently have the capability to distinguish between upper- and lower-case in the same fashion. In this case, you might need something like this (which I have thrown together fairly quickly without taking the time to make it pretty):

>>> def split_at_uppercase(text):
    result = []
    new_word = []
    for char in text:
        if char.isupper():
            if new_word:
                result.append(''.join(new_word))
            new_word = []
            new_word.append(char)
        elif new_word and char == ' ':  # in more complicate scenarios, may need to use regex for white space
            result.append(''.join(new_word))
            new_word = []
        elif char != ' ':
            new_word.append(char)
    else:
        result.append(''.join(new_word))
    return result

>>> t = 'καὶ τοῦΠιλάτου εἰςἹεροσόλυμα'
>>> split_at_uppercase(t)
['καὶ', 'τοῦ', 'Πιλάτου', 'εἰς', 'Ἱεροσόλυμα']

Then, as @acarlon has suggested, you will start needing to check substrings against a dictionary (such as PyEnchant). But even then, you will find 'meaning' where no one may have meant to impart it. And further, you will discover words that do not interest you (such as prod).

Answer 2

I would suggest:

1. Extract all words (sequences of letters - this will include compound words such as ProductTitle), i.e. strip out everything that is not a letter into groups. Regex something like ([a-zA-Z])+?). +? means non-greedy.
2. For each word look up the word in an english dictionary database. If there is a match then move on to the next word. If not go to step 3.
3. If no match, then try to find multiple words in the compound word, e.g. ProductTitle -> Product Title. This is not a trivial task, requires a decision tree type operation and the ability to roll back. Have a look at tries and at this answer.