How to plot the pricipal vectors of each variable after performing PCA?

Question

My question mainly comes from this post :https://stats.stackexchange.com/questions/53/pca-on-correlation-or-covariance

In the article, the author plotted the vector direction and length of each variable. Based on my understanding, after performing PCA. All we get are the eigenvectors and eigenvalues. For a dataset which has a dimension M x N, each eigenvalue should be a vector as 1 x N. So, my question is maybe the length of the vector is the eigenvalue, but how to find the direction of the vector for each variable mathematical? And what is the physical meaning of the length of the vector?

Also, if it is possible, can I do similar work with scikit PCA function in python?

Thanks!

https://jakevdp.github.io/PythonDataScienceHandbook/05.09-principal-component-analysis.html — hellpanderr, Aug 03 '19 at 22:27

seralouk · Accepted Answer · 2020-11-05T10:32:02.423

This plot is called biplot and it is very useful to understand the PCA results. The length of the vectors it is just the values that each feature/variable has on each Principal Component aka PCA loadings.

Example:

These loadings as accessible through print(pca.components_). Using the Iris Dataset the loadings are:

  [[ 0.52106591, -0.26934744,  0.5804131 ,  0.56485654],
   [ 0.37741762,  0.92329566,  0.02449161,  0.06694199],
   [-0.71956635,  0.24438178,  0.14212637,  0.63427274],
   [-0.26128628,  0.12350962,  0.80144925, -0.52359713]])

Here, each row is one PC and each column corresponds to one variable/feature. So feature/variable 1, has a value 0.52106591 on the PC1 and 0.37741762 on the PC2. These are the values used to plot the vectors that you saw in the biplot. See below the coordinates of Var1. It's exactly those (above) values !!

Finally, to create this plot in python you can use this using sklearn:

import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

iris = datasets.load_iris()
X = iris.data
y = iris.target

#In general it is a good idea to scale the data
scaler = StandardScaler()
scaler.fit(X)
X=scaler.transform(X)

pca = PCA()
pca.fit(X,y)
x_new = pca.transform(X)   

def myplot(score,coeff,labels=None):
    xs = score[:,0]
    ys = score[:,1]
    n = coeff.shape[0]

    plt.scatter(xs ,ys, c = y) #without scaling
    for i in range(n):
        plt.arrow(0, 0, coeff[i,0], coeff[i,1],color = 'r',alpha = 0.5)
        if labels is None:
            plt.text(coeff[i,0]* 1.15, coeff[i,1] * 1.15, "Var"+str(i+1), color = 'g', ha = 'center', va = 'center')
        else:
            plt.text(coeff[i,0]* 1.15, coeff[i,1] * 1.15, labels[i], color = 'g', ha = 'center', va = 'center')

plt.xlabel("PC{}".format(1))
plt.ylabel("PC{}".format(2))
plt.grid()

#Call the function. 
myplot(x_new[:,0:2], pca.components_.T) 
plt.show()

See also this post: https://stackoverflow.com/a/50845697/5025009

and

https://towardsdatascience.com/pca-clearly-explained-how-when-why-to-use-it-and-feature-importance-a-guide-in-python-7c274582c37e?source=friends_link&sk=65bf5440e444c24aff192fedf9f8b64f

Thanks, your answer and the link are really helpful for understanding. — Xudong, Aug 04 '19 at 17:18
By the way, do you see the link which is posted by hellpanderr? The link said they use pca.explained_variance_ to determine the length of the vector. Is it wrong? Or because of some other reasons? — Xudong, Aug 04 '19 at 17:27
they use the "explained variance" to define the squared-length of the vector. it's equivalent — seralouk, Aug 04 '19 at 17:54
where are these values (>Here, each row is one PC and each column corresponds to one variable/feature. So feature/variable 1, has a value 0.5223 on the PC1 and 0.3723 on the PC2.) from? — malocho, May 04 '20 at 11:08
these can be printed using `print(pca.components_)`. I state this in my answer explicitly. Read again my naswer. — seralouk, May 04 '20 at 11:17
@seralouk yes, you wrote it but the values of pca.components_ do not consist of the values which you have in your description. The first value has 0.52106591 and -0.26934744. — malocho, May 04 '20 at 18:12
The index in `pca.components_` is wrong in this answer. It should be `pca.components_[c, f]`, where `c` is `0`/`1` (x or y in the plot) and `f` is the feature number. — SOFe, Nov 05 '20 at 07:04
No, it's not wrong. it was just a transpose missing. Call it as: `myplot(x_new[:,0:2], pca.components_.T) ` — seralouk, Nov 05 '20 at 10:33

erdogant · Answer 2 · 2022-03-06T23:06:35.293

Try the pca library. This will plot the explained variance, and create a biplot.

pip install pca

A small example:

from pca import pca

# Initialize to reduce the data up to the number of componentes that explains 95% of the variance.
model = pca(n_components=0.95)

# Or reduce the data towards 2 PCs
model = pca(n_components=2)

# Load example dataset
import pandas as pd
import sklearn
from sklearn.datasets import load_iris
X = pd.DataFrame(data=load_iris().data, columns=load_iris().feature_names, index=load_iris().target)

# Fit transform
results = model.fit_transform(X)

# Plot explained variance
fig, ax = model.plot()

explained variance PCs

# Scatter first 2 PCs
fig, ax = model.scatter()

# Make biplot with the number of features
fig, ax = model.biplot(n_feat=4)

PCA biplot

The results is a dict containing many statistics of the PCs, loadings etc.

How to plot the pricipal vectors of each variable after performing PCA?

2 Answers2