going from numpy array to a pandas dataframe changes values

Question

consider the following code:

dog = np.random.rand(10, 10)
frog = pd.DataFrame(dog, columns = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'])
from sklearn.preprocessing import StandardScaler
slog = StandardScaler()
mog = slog.fit_transform(frog.values)
frog[frog.columns] = mog

OK, now we should have a dataframe whose values should be the standard-scaled array. But:

frog.describe()

gives:

[![describe the dataframe][1]][1]

Note that the standard deviation is 1.05

While

np.std(mog, axis=0)

Gives the expected:

array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

What gives?

I am not sure why the image is not showing up, but the standard deviation is 1.05 in all columns. — Igor Rivin, May 24 '20 at 19:53
Partial duplicate: https://stackoverflow.com/questions/27600207/why-does-numpy-std-give-a-different-result-to-matlab-std — Andras Deak -- Слава Україні, May 24 '20 at 20:13

score 1 · Accepted Answer · answered May 24 '20 at 20:09

The standard deviation computed by the describe method uses the sample standard deviation, while StandardScaler uses the population standard deviation. The only difference between the two is whether the sum of the squared differences from the mean is divided by n-1 (for the sample st. dev.) or n (for the pop. std. dev.).

numpy.std computes the population st. dev. by default, but you can use it to compute the sample st. dev. by adding the argument ddof=1, and the result agrees with the values computed by describe:

In [54]: np.std(mog, axis=0, ddof=1)
Out[54]: 
array([1.05409255, 1.05409255, 1.05409255, 1.05409255, 1.05409255,
       1.05409255, 1.05409255, 1.05409255, 1.05409255, 1.05409255])

going from numpy array to a pandas dataframe changes values

1 Answers1