Case Insensitive replacement (mapping)

Question

I posted a "part 1" question that got me to the answer of the function I needed here but thought that this warranted its own question. If not, I will remove.

I want to apply a function to a dataframe that replaces the Full state name to the Abbreviation (New York -> NY). However I noticed in my dataset that if a state was Capitalized it obviously would not match the dicitonary. I tried to work around it, but can't seem to crack the code:

import pandas as pd
import numpy as np
dfp = pd.DataFrame({'A' : [np.NaN,np.NaN,3,4,5,5,3,1,5,np.NaN], 
                    'B' : [1,0,3,5,0,0,np.NaN,9,0,0], 
                    'C' : ['Pharmacy of IDAHO','NY Pharma','NJ Pharmacy','Idaho Rx','CA Herbals','Florida Pharma','AK RX','Ohio Drugs','PA Rx','USA Pharma'], 
                    'D' : [123456,123456,1234567,12345678,12345,12345,12345678,123456789,1234567,np.NaN],
                    'E' : ['Assign','Unassign','Assign','Ugly','Appreciate','Undo','Assign','Unicycle','Assign','Unicorn',]})

import us
statez = us.states.mapping('abbr', 'name')
inv_map = {v: k for k, v in statez.items()}

def replace_states(company):
    # find all states that exist in the string
    state_found = filter(lambda state: state.lower() in company.lower(), statez.values())

    # replace each state with its abbreviation
    for state in state_found:
        print(state, inv_map[state])
        company = company.replace(state, inv_map[state])
        print("---" , company)

    # return the modified string (or original if no states were found)
    return company

dfp['C'] = dfp['C'].map(replace_states)

output: notice the lack of change in " Pharmacy of IDAHO"

Idaho ID
--- Pharmacy of IDAHO
Idaho ID
--- ID Rx
Florida FL
--- FL Pharma
Ohio OH
--- OH Drug

is there a way to make this function case insensitive?

Answer 1

I would find the index for it, then use that to replace it regardless of case:

# replace each state with its abbreviation
    for state in state_found:
        print(state, inv_map[state])
        index = company.lower().find(state.lower())
        company = company.replace(company[index:index + len(state)], inv_map[state])
        print("---" , company)

This preserves the case for all other parts of the string.

Answer 2

Replacing states names with their abbreviations (case-insensitive vectorized solution):

t1 = dfp.C.str.split(expand=True)
t2 = t1.stack().str.title().map(inv_map).unstack()
t1[t2.notnull()] = t2
dfp['new'] = t1.stack().groupby(level=0).agg(' '.join)

Result:

In [152]: x
Out[152]:
     A    B                  C            D           E             new
0  NaN  1.0  Pharmacy of IDAHO     123456.0      Assign  Pharmacy of ID
1  NaN  0.0          NY Pharma     123456.0    Unassign       NY Pharma
2  3.0  3.0        NJ Pharmacy    1234567.0      Assign     NJ Pharmacy
3  4.0  5.0           Idaho Rx   12345678.0        Ugly           ID Rx
4  5.0  0.0         CA Herbals      12345.0  Appreciate      CA Herbals
5  5.0  0.0     Florida Pharma      12345.0        Undo       FL Pharma
6  3.0  NaN              AK RX   12345678.0      Assign           AK RX
7  1.0  9.0         Ohio Drugs  123456789.0    Unicycle        OH Drugs
8  5.0  0.0              PA Rx    1234567.0      Assign           PA Rx
9  NaN  0.0         USA Pharma          NaN     Unicorn      USA Pharma

Explanation:

In [155]: t1 = dfp.C.str.split(expand=True)

In [156]: t1
Out[156]:
          0         1      2
0  Pharmacy        of  IDAHO
1        NY    Pharma   None
2        NJ  Pharmacy   None
3     Idaho        Rx   None
4        CA   Herbals   None
5   Florida    Pharma   None
6        AK        RX   None
7      Ohio     Drugs   None
8        PA        Rx   None
9       USA    Pharma   None

In [157]: t2 = t1.stack().str.title().map(inv_map).unstack()

In [158]: t2
Out[158]:
     0    1     2
0  NaN  NaN    ID
1  NaN  NaN  None
2  NaN  NaN  None
3   ID  NaN  None
4  NaN  NaN  None
5   FL  NaN  None
6  NaN  NaN  None
7   OH  NaN  None
8  NaN  NaN  None
9  NaN  NaN  None

In [159]: t1[t2.notnull()] = t2

In [160]: t1
Out[160]:
          0         1     2
0  Pharmacy        of    ID
1        NY    Pharma  None
2        NJ  Pharmacy  None
3        ID        Rx  None
4        CA   Herbals  None
5        FL    Pharma  None
6        AK        RX  None
7        OH     Drugs  None
8        PA        Rx  None
9       USA    Pharma  None

---

Replacing states abbreviations with their names (case-insensitive vectorized solution):

In [88]: dfp['state'] = dfp.C.str.extract(r'\b([A-Z]{2})\b', expand=False)

In [89]: dfp
Out[89]:
     A    B                  C            D           E state
0  NaN  1.0  Pharmacy of IDAHO     123456.0      Assign   NaN
1  NaN  0.0          NY Pharma     123456.0    Unassign    NY
2  3.0  3.0        NJ Pharmacy    1234567.0      Assign    NJ
3  4.0  5.0           Idaho Rx   12345678.0        Ugly   NaN
4  5.0  0.0         CA Herbals      12345.0  Appreciate    CA
5  5.0  0.0     Florida Pharma      12345.0        Undo   NaN
6  3.0  NaN              AK RX   12345678.0      Assign    AK
7  1.0  9.0         Ohio Drugs  123456789.0    Unicycle   NaN
8  5.0  0.0              PA Rx    1234567.0      Assign    PA
9  NaN  0.0         USA Pharma          NaN     Unicorn   NaN

In [90]: dfp.C = dfp.C.replace(dfp.state.tolist(),
                               dfp.state.map(statez).tolist(),
                               regex=True)

In [91]: dfp
Out[91]:
     A    B                    C            D           E state
0  NaN  1.0    Pharmacy of IDAHO     123456.0      Assign   NaN
1  NaN  0.0      New York Pharma     123456.0    Unassign    NY
2  3.0  3.0  New Jersey Pharmacy    1234567.0      Assign    NJ
3  4.0  5.0             Idaho Rx   12345678.0        Ugly   NaN
4  5.0  0.0   California Herbals      12345.0  Appreciate    CA
5  5.0  0.0       Florida Pharma      12345.0        Undo   NaN
6  3.0  NaN            Alaska RX   12345678.0      Assign    AK
7  1.0  9.0           Ohio Drugs  123456789.0    Unicycle   NaN
8  5.0  0.0      Pennsylvania Rx    1234567.0      Assign    PA
9  NaN  0.0           USA Pharma          NaN     Unicorn   NaN