How to whiten matrix in PCA

Question

I'm working with Python and I've implemented the PCA using this tutorial.

Everything works great, I got the Covariance I did a successful transform, brought it make to the original dimensions not problem.

But how do I perform whitening? I tried dividing the eigenvectors by the eigenvalues:

S, V = numpy.linalg.eig(cov)
V = V / S[:, numpy.newaxis]

and used V to transform the data but this led to weird data values. Could someone please shred some light on this?

You might want to try a more specific mathematical venue, perhaps a mailing list associated with numpy or scikits. — Thomas K, Jul 04 '11 at 18:37

ali_m · Accepted Answer · 2017-06-12T21:37:41.213

26

Here's a numpy implementation of some Matlab code for matrix whitening I got from here.

import numpy as np

def whiten(X,fudge=1E-18):

   # the matrix X should be observations-by-components

   # get the covariance matrix
   Xcov = np.dot(X.T,X)

   # eigenvalue decomposition of the covariance matrix
   d, V = np.linalg.eigh(Xcov)

   # a fudge factor can be used so that eigenvectors associated with
   # small eigenvalues do not get overamplified.
   D = np.diag(1. / np.sqrt(d+fudge))

   # whitening matrix
   W = np.dot(np.dot(V, D), V.T)

   # multiply by the whitening matrix
   X_white = np.dot(X, W)

   return X_white, W

You can also whiten a matrix using SVD:

def svd_whiten(X):

    U, s, Vt = np.linalg.svd(X, full_matrices=False)

    # U and Vt are the singular matrices, and s contains the singular values.
    # Since the rows of both U and Vt are orthonormal vectors, then U * Vt
    # will be white
    X_white = np.dot(U, Vt)

    return X_white

The second way is a bit slower, but probably more numerically stable.

edited Jun 12 '17 at 21:37

answered Jul 04 '12 at 23:01

ali_m

71,714
23
223
298

Thanks! Shouldn't the svd be performed on the covariance matrix of X? – Ran Jan 04 '15 at 21:45
@Ran I think you are confusing SVD with eigendecomposition. Using the SVD method you don't explicitly compute the covariance matrix beforehand - the columns of `U` will contain the eigenvectors of `X * X.T`, and the rows of `Vt` contain the eigenvectors of `X.T * X`. Since the rows of `U` and `Vt` are orthonormal vectors, the covariance matrix of `U.dot(Vt)` will be the identity. – ali_m Jan 05 '15 at 11:46
All the other examples I saw perform the svd on the covariance matrix, e.g.https://gist.github.com/duschendestroyer/5170087 . – Ran Jan 09 '15 at 12:48
@Ran The example you just linked to shows [ZCA whitening](http://ufldl.stanford.edu/wiki/index.php/Whitening), which is one of many different ways to whiten a matrix. For any orthogonal matrix `R`, `R * X_white` will also have identity covariance. In ZCA, `R` is chosen to be `U` (i.e. the eigenvectors of `X * X.T`). This particular transformation results in whitened data that is as close as possible to `X` (in the least-squares sense). If you just want whitened data you can compute `X_white` as above (take a look at the values in `X_white.T * X_white` if you don't believe me). – ali_m Jan 12 '15 at 11:11
beware people that this is true just for symmetric matrices.. :) – asdf Jun 06 '17 at 16:45
@asdf The SVD approach will also work for non-square inputs if you pass `full_matrices=False` to `np.linalg.svd` (see my update). Equivalently, you could replace the singular values with an observations-by-components identity matrix, e.g. `U.dot(np.eye(*X.shape)).dot(Vt)`, but it's cheaper to simply avoid calculating the full matrices for `U` and `Vt`. – ali_m Jun 12 '17 at 21:49
The result of the first and second solution is different for a 3*2 matrix like ([1,2],[2,4],[3,6]). SVD removes linearity but the first one doesn't – Ahmad Dec 30 '19 at 08:10
2

Hi there, I think your covariance matrix computations assumes that the data has already been centered at zero, right? – Jonasson Feb 03 '20 at 09:54
Dear Ali_m, is this method suitable for large number of data for example 400 number? – Sik Sik Sep 18 '21 at 17:45

score 15 · Answer 2 · edited May 25 '18 at 23:59

15

If you use python's scikit-learn library for this, you can just set the inbuilt parameter

from sklearn.decomposition import PCA
pca = PCA(whiten=True)
whitened = pca.fit_transform(X)

check the documentation.

edited May 25 '18 at 23:59

Arya McCarthy

8,554
4
34
56

answered Jun 28 '16 at 12:55

Shubham Bansal

388
5
8

score 1 · Answer 3 · answered Jul 06 '11 at 21:54

1

I think you need to transpose V and take the square root of S. So the formula is

matrix_to_multiply_with_data = transpose( v ) * s^(-1/2 )

answered Jul 06 '11 at 21:54

Krish

1,747
14
19

score 0 · Answer 4 · answered Dec 05 '21 at 17:44

0

Use ZCA mapping instead

function [Xw] = whiten(X)
  % Compute and apply the ZCA mapping
  mu_X = mean(X, 1);
  X = bsxfun(@minus, X, mu_X);
  Xw = X / sqrtm(cov(X));
end

answered Dec 05 '21 at 17:44

euraad

2,467
5
30
51

what language is this? – thistleknot Dec 07 '21 at 01:06
1

@thistleknot This is not.... the greatest language in the world, no ....This is just MATLAB. – euraad Dec 08 '21 at 08:16

How to whiten matrix in PCA

4 Answers4

Linked