Pandas apply & map to every element of every column

Question

How to apply a custom function to every element of every column if its the value is not null?

Lets say I have a data frame of 10 columns, out of which I want to apply a lower() function to every element of just 4 columns if pd.notnull(x), else just keep none as value.

I tried to use like this,

s.apply(lambda x: change_to_lowercase(x), axis = 1)

def change_to_lowercase(s):

    s['A'] =  s['A'].map(lambda x: x.lower() if pd.notnull(x) else x)
    s['B'] = s['B'].map(lambda x: x.lower() if pd.notnull(x) else x)
    s['C'] = s['C'].map(lambda x: x.lower() if pd.notnull(x) else x)
    s['D'] = s['D'].map(lambda x: x.lower() if pd.notnull(x) else x)
    return s

But since my columns are mixed datatype(which is NaN as float, rest as unicode). This is throwing me an error -

float has no attribute map.

How to get rid of this error?

score 5 · Answer 1 · answered Jun 15 '17 at 05:11

I think you need DataFrame.applymap because working elementwise:

L = [[1.5, 'Test', np.nan, 2], ['Test', np.nan, 2,'TEST'], ['Test', np.nan,1.5,  2]]
df = pd.DataFrame(L, columns=list('abcd'))
print (df)

      a     b    c     d
0   1.5  Test  NaN     2
1  Test   NaN  2.0  TEST
2  Test   NaN  1.5     2

cols = ['a','b']
#for python 2 change str to basestring
df[cols] = df[cols].applymap(lambda x: x.lower() if isinstance(x, str) else x)
print (df)
      a     b    c     d
0   1.5  test  NaN     2
1  test   NaN  2.0  TEST
2  test   NaN  1.5     2

Claudiu Creanga · Accepted Answer · 2017-06-15T03:54:09.983

You are trying to map a Series and then in lambda you take the entire row.

You should also check for integers, floats etc that don't have a method .lower(). So the best is to check if it is a string, not just if it is not a notnull, in my opinion.

This works:

s = pd.DataFrame([{'A': 1.5, 'B':"Test", 'C': np.nan, 'D':2}])
s

        A   B   C   D
0   1.5 Test    NaN 2



s1 = s.apply(lambda x: x[0].lower() if isinstance(x[0], basestring) else x[0]).copy()

s1
    A     1.5
    B    test
    C     NaN
    D       2
    dtype: object

For python 3 to check if string isinstance(x[0], str)

To be able to select columns:

s1 = pd.DataFrame()
columns = ["A", "B"]
for column in columns:
    s1[column] = s[column].apply(lambda x: x.lower() if isinstance(x, str) else x).copy()
s1

    A   B
0   1.5 test

Thank you. That makes sense. How do I apply this only for some columns in my data frame? — ds_user, Jun 15 '17 at 03:37

Pandas apply & map to every element of every column

2 Answers2