0

I have a dataframe next_train with weekly data for many players (80,000 players observed through 4 weeks, total of 320,000 observations) and a dictionary players containing a binary variable for some of the players (say 10,000). I want to add this binary variable to the dataframe next_train (if a player is not in the dictionary players, I set the variable equal to zero). This is how I'm doing it:

next_train = pd.read_csv()
# ... calculate dictionary 'players' ...
next_train['variable'] = 0
for player in players:
    next_train.loc[next_train['id_of_player'] == player, 'variable'] = players[player]

However the for loop takes ages to complete, and I don't understand why. It looks like the task is to perform binary search for the value player in my dataframe for 10,000 times (size of the players dictionary), but the execution time is several minutes. Is there any efficient way to do this task?

Alexandr Kapshuk
  • 1,380
  • 2
  • 13
  • 29

2 Answers2

1

You should use map instead of slicing, that will be way faster:

next_train['variable'] = next_train.id_of_player.map(players)

As you want 0 in the other rows, you can then use fillna:

next_train.variable.fillna(0,inplace = True)

Moreover, if your dictionnary only contains boolean values, you might want to redefine the type of variable column to take less space. So you end with this piece of code:

next_train['variable'] = next_train.id_of_player.map(players).fillna(0).astype(int)
ysearka
  • 3,805
  • 5
  • 20
  • 41
1

Use map and fillna:

next_train['variable'] = next_train['id_of_player'].map(players).fillna(0)

This creates a new column by applying the dictionary on the player ids and then fills all empty values with 0.

Shaido
  • 27,497
  • 23
  • 70
  • 73