3

I am trying to count the number of changes of value in each column in a data frame in pandas. The code I have works great except for NaNs: if a column contains two subsequent NaNs, it is counted as a change of value, which I don't want. How can I avoid that?

I do as follows (thanks to unutbu's answer):

import pandas as pd
import numpy as np

frame = pd.DataFrame({
    'time':[1234567000 , np.NaN, np.NaN],
    'X1':[96.32,96.01,96.05],
    'X2':[23.88,23.96,23.96]
},columns=['time','X1','X2']) 

print(frame)

changes = (frame.diff(axis=0) != 0).sum(axis=0)
print(changes)

changes = (frame != frame.shift(axis=0)).sum(axis=0)
print(changes)

returns:

           time     X1     X2
0  1.234567e+09  96.32  23.88
1           NaN  96.01  23.96
2           NaN  96.05  23.96

time    3
X1      3
X2      2
dtype: int64

time    3
X1      3
X2      2
dtype: int64

Instead, the results should be (notice the change in the time column):

time    2
X1      3
X2      2
dtype: int64
Franck Dernoncourt
  • 77,520
  • 72
  • 342
  • 501

1 Answers1

3
change = (frame.fillna(0).diff() != 0).sum()

Output:

time    2
X1      3
X2      2
dtype: int64

NaN are "truthy". Change NaN to zero then evaluate.

nan - nan = nan

nan != 0  = True

fillna(0)

0 - 0 = 0

0 != 0 = False
Scott Boston
  • 147,308
  • 15
  • 139
  • 187