Python Pandas Column and Fuzzy Match + Replace

Question

Intro

Hello, I'm working on a project that requires me to replace dictionary keys within a pandas column of text with values - but with potential misspellings. Specifically I am matching names within a pandas column of text and replacing them with "First Name". For example, I would be replacing "tommy" with "First Name".

However, I realize there's the issue of misspelled names and text within the column of strings that won't be replaced by my dictionary. For example 'tommmmy" has extra m's and is not a first name within my dictionary.

#Create df 
d = {'message' : pd.Series(['awesome', 'my name is tommmy , please help with...', 'hi tommy , we understand your quest...'])}
names = ["tommy", "zelda", "marcon"]

#create dict 
namesdict = {r'(^|\s){}($|\s)'.format(el): r'\1FirstName\2' for el in names}

#replace 
d['message'].replace(namesdict, regex = True)



  #output 
    Out: 
0                                       awesome
1    my name is tommmy , please help with...
2    hi FirstName , we understand your quest...
dtype: object

so "tommmy" doesn't match to "tommy" in the -> I need to deal with misspellings. I thought about trying to do this prior to the actual dictionary key and value replacement, like scan through the pandas data frame and replace the words within the column of strings ("messages") with the appropriate name. I've seen a similar approach using an index on specific strings like this one

but how do you match and replace words within the sentences within a pandas df, using a list of correct spelling? Can I do this within the df.series replace argument? Should I stick with a regex string replace?*

Any suggestions appreciated.

Update , trying Yannis's answer

I'm trying Yannis's answer but I need to use a list from an outside source, specifically the US census of first names for matching. But it's not matching on the whole names with the string I download.

d = {'message' : pd.Series(['awesome', 'my name is tommy , please help with...', 'hi tommy , we understand your quest...'])}

import requests 
r = requests.get('http://deron.meranda.us/data/census-derived-all-first.txt')

#US Census first names (5000 +) 
firstnamelist = re.findall(r'\n(.*?)\s', r.text, re.DOTALL)


#turn list to string, force lower case
fnstring = ', '.join('"{0}"'.format(w) for w in firstnamelist )
fnstring  = ','.join(firstnamelist)
fnstring  = (fnstring.lower())


##turn to list, prepare it so it matches the name preceded by either the beginning of the string or whitespace.  
names = [x.strip() for x in fnstring.split(',')]




#import jellyfish 
import difflib 


def best_match(tokens, names):
    for i,t in enumerate(tokens):
        closest = difflib.get_close_matches(t, names, n=1)
        if len(closest) > 0:
            return i, closest[0]
    return None

def fuzzy_replace(x, y):
    
    names = y # just a simple replacement list
    tokens = x.split()
    res = best_match(tokens, y)
    if res is not None:
        pos, replacement = res
        tokens[pos] = "FirstName"
        return u" ".join(tokens)
    return x

d["message"].apply(lambda x: fuzzy_replace(x, names))

Results in:

Out: 
0                                        FirstName
1    FirstName name is tommy , please help with...
2    FirstName tommy , we understand your quest...

But if I use a smaller list like this it works:

names = ["tommy", "caitlyn", "kat", "al", "hope"]
d["message"].apply(lambda x: fuzzy_replace(x, names))

Is it something with the longer list of names that's causing a problem?

Answer 1

Edit:

Changed my solution to use difflib. The core idea is to tokenize your input text and match each token against a list of names. If best_match finds a match then it reports the position (and the best matching string), so then you can replace the token with "FirstName" or anything you want. See the complete example below:

import pandas as pd
import difflib

df = pd.DataFrame(data=[(0,"my name is tommmy , please help with"), (1, "hi FirstName , we understand your quest")], columns=["A", "message"])

def best_match(tokens, names):
    for i,t in enumerate(tokens):
        closest = difflib.get_close_matches(t, names, n=1)
        if len(closest) > 0:
            return i, closest[0]
    return None

def fuzzy_replace(x):
    names = ["tommy", "john"] # just a simple replacement list
    tokens = x.split()
    res = best_match(tokens, names)
    if res is not None:
        pos, replacement = res
        tokens[pos] = "FirstName"
        return u" ".join(tokens)
    return x

df.message.apply(lambda x: fuzzy_replace(x))

And the output you should get is the following

0    my name is FirstName , please help with
1    hi FirstName , we understand your quest
Name: message, dtype: object

Edit 2

After the discussion, I decided to have another go, using NLTK for parts of speech tagging and run the fuzzy matching only for the NNP tags (proper nouns) against the name list. The problem is that sometimes the tagger doesn't get the tag right, e.g. "Hi" might be also tagged as proper noun. However if the list of names are lowercased then get_close_matches doesn't match Hi against a name but matches all other names. I recommend that df["message"] is not lowercased to increase the chances that NLTK tags the names properly. One can also play with StanfordNER but nothing will work 100%. Here is the code:

import pandas as pd
import difflib
from nltk import pos_tag, wordpunct_tokenize
import requests 
import re

r = requests.get('http://deron.meranda.us/data/census-derived-all-first.txt')

# US Census first names (5000 +) 
firstnamelist = re.findall(r'\n(.*?)\s', r.text, re.DOTALL)

# turn list to string, force lower case
# simplified things here
names = [w.lower() for w in firstnamelist]


df = pd.DataFrame(data=[(0,"My name is Tommmy, please help with"), 
                        (1, "Hi Tommy , we understand your question"),
                        (2, "I don't talk to Johhn any longer"),
                        (3, 'Michale says this is stupid')
                       ], columns=["A", "message"])

def match_names(token, tag):
    print token, tag
    if tag == "NNP":
        best_match = difflib.get_close_matches(token, names, n=1)
        if len(best_match) > 0:
            return "FirstName" # or best_match[0] if you want to return the name found
        else:
            return token
    else:
        return token

def fuzzy_replace(x):
    tokens = wordpunct_tokenize(x)
    pos_tokens = pos_tag(tokens)
    # Every token is a tuple (token, tag)
    result = [match_names(token, tag) for token, tag in pos_tokens]
    x = u" ".join(result)
    return x

df['message'].apply(lambda x: fuzzy_replace(x))

And I get in the output:

0       My name is FirstName , please help with
1    Hi FirstName , we understand your question
2        I don ' t talk to FirstName any longer
3                 FirstName says this is stupid
Name: message, dtype: object