2

I am trying to remap data from dictionary to the respective values in dataframe and I am successful in it.

It is fine when I am using few thousands of data but gets radically slow and sometimes never ending if I try to do it with few millions of data.

Here is the code I am using and works fine for less data: 

def remap(data,dict_labels):
    for field,values in dict_labels.items():
        data.replace({field:values},inplace=True)
    print("DONE")
    return data
dataframe1 = remap(df,dataDict)

Is there any efficient way of using something else for the same task so that it is faster for large data?

jpp
  • 159,742
  • 34
  • 281
  • 339
  • 1
    Why do you need the for loop? Isn't what you are doing the same as just `data.replace(dict_labels, inplace=True)`? – jdehesa Apr 06 '18 at 11:07
  • Possible duplicate of https://stackoverflow.com/questions/41985566/pandas-replace-dictionary-slowness – cdwoelk Apr 06 '18 at 11:11

1 Answers1

2

You should use pd.DataFrame.applymap for this:

dataframe1 = df.applymap(dict_labels.get).fillna(df)

Note we use fillna to recover unmapped values.

As per the documentation:

DataFrame.applymap(func)

Apply a function to a DataFrame that is intended to operate elementwise, i.e. like doing map(func, series) for each series in the DataFrame

Performance note: Replace values in a pandas series via dictionary efficiently

jpp
  • 159,742
  • 34
  • 281
  • 339