7

I'm trying to use KernelPCA for reducing the dimensionality of a dataset to 2D (both for visualization purposes and for further data analysis).

I experimented computing KernelPCA using a RBF kernel at various values of Gamma, but the result is unstable:

anim

(each frame is a slightly different value of Gamma, where Gamma is varying continuously from 0 to 1)

Looks like it is not deterministic.

Is there a way to stabilize it/make it deterministic?

Code used to generate transformed data:

def pca(X, gamma1):
    kpca = KernelPCA(kernel="rbf", fit_inverse_transform=True, gamma=gamma1)
    X_kpca = kpca.fit_transform(X)
    #X_back = kpca.inverse_transform(X_kpca)
    return X_kpca
fferri
  • 18,285
  • 5
  • 46
  • 95

2 Answers2

3

KernelPCA should be deterministic and evolve continuously with gamma.
It is different from RBFSampler that does have built-in randomness in order to provide an efficient (more scalable) approximation of the RBF kernel.

However what can change in KernelPCA is the order of the principal components: in scikit-learn they are returned sorted in order of descending eigenvalue, so if you have 2 eigenvalues close to each other it could be that the order changes with gamma.

My guess (from the gif) is that this is what is happening here: the axes along which you are plotting are not constant so your data seems to jump around.

Could you provide the code you used to produce the gif?

I'm guessing it is a plot of the data points along the 2 first principal components but it would help to see how you produced it.

You could try to further inspect it by looking at the values of kpca.alphas_ (the eigenvectors) for each value of gamma.

Hope this makes sense.

EDIT: As you noted it looks like the points are reflected against the axis, the most plausible explanation is that one of the eigenvector flips sign (note this does not affect the eigenvalue).

I put in a simple gist to reproduce the issue (you'll need a Jupyter notebook to run it). You can see the sign-flipping when you change the value of gamma.

As a complement note that this kind of discrepancy happens only because you fit several times the KernelPCA object several times. Once you settled with a particular gamma value and you've fit kpca once you can call transform several times and get consistent results. For the classical PCA the docs mention that:

Due to implementation subtleties of the Singular Value Decomposition (SVD), which is used in this implementation, running fit twice on the same matrix can lead to principal components with signs flipped (change in direction). For this reason, it is important to always use the same estimator object to transform data in a consistent fashion.

I don't know about the behavior of a single KernelPCA object that you would fit several times (I did not find anything relevant in the docs).

It does not apply to your case though as you have to fit the object with several gamma values.

ldirer
  • 6,606
  • 3
  • 24
  • 30
  • Yes, the plot is the first two principal components – fferri Jul 06 '15 at 10:07
  • 1
    Ok. Do you see how the order of the components may affect your plot? Say your first 2 components are e1 and e2, with associated eigenvalues a1 and a2. For gamma = 0.5, a1 > a2. So KernelPCA returns the eigenvectors in order e1, e2, e3... For gamma = 0.6, a2 > a1. Now you are getting e2, e1, e3... and your axes on the plot are swapped. – ldirer Jul 06 '15 at 11:25
  • Seems more like the eigenvalues flip sign. If swapping axes (x with y) I should observe the points *transpose*. Instead the points *reflect* in x or y. Don't you agree? – fferri Jul 06 '15 at 15:12
  • Yes I kind of agree with you ;). It is true that we see something that looks like a reflection (actually I see 2 different reflections). However negative eigenvalues are not an option in PCA, hence I'd rather fool myself and believe it's a transposition... Could you give us the eigenvectors/eigenvalues at each stage (kpca.alphas_ and kpca.lambdas_ respectively)? – ldirer Jul 06 '15 at 15:34
  • Makes sense. You can get only the 3 first ones that should be enough to see if something is happening here. They are sorted with highest lambda first in scikit-learn. – ldirer Jul 06 '15 at 16:14
  • First values of lambdas are for example 6.96279080e+02 6.79270038e+02 6.58483207e+02 6.50819318e+02 6.29440222e+02 6.13970213e+02 6.10199451e+02 ... – fferri Jul 06 '15 at 16:16
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/82517/discussion-between-user3914041-and-mescalinum). – ldirer Jul 06 '15 at 16:16
  • 1
    Ok you are right about the sign flipping, but it is not the **eigenvalues** that flip sign. It is just the eigenvector that changes to - 1 * itself. – ldirer Jul 06 '15 at 16:56
1

So... I can't give you a definitive answer on why KernelPCA is not deterministic. The behavior resembles the differences I've observed between the results of PCA and RandomizedPCA. PCA is deterministic, but RandomizedPCA is not, and sometimes the eigenvectors are flipped in sign relative to the PCA eigenvectors.

That leads me to my vague idea of how you might get more deterministic results....maybe. Use RBFSampler with a fixed seed:

def pca(X, gamma1):
    kernvals = RBFSampler(gamma=gamma1, random_state=0).fit_transform(X)
    kpca = PCA().fit_transform(X)
    X_kpca = kpca.fit_transform(X)
    return X_kpca
Andreus
  • 2,437
  • 14
  • 22