Suppose your points are Q1..Qn. We can define a line by a point P on the line and a unit vector u that points along the line.
In any dimension, we can find the line as below.
Given a point Q the closest point on the line to Q is then
Q^ = P + u'*(Q-P)*u
The squared distance of Q from the line is the distance squared between Q and Q^
dsq = || Q-Q^||^2 = ||(Q-P)||^2 - (u'*(Q-P))^2
One approach to finding the best fit line is to look for the line that minimises the mean squared distances of the points from the line, ie to minimise
E = Sum{ j | ||(Q[j]-P)||^2 - (u'*(Q[j]-P))^2 }/n
Some rather tedious algebra shows that this means that
P = Sum{ j | Q[j]}/n -- ie the mean of the Qs
u the eigenvector of C corresponding to the largest eigenvalue of C, where
C = Sum{ j | (Q[j]-P)*(Q[j]-P)'} -- the covariance of the Qs
So the drill to find P and u is:
Compute P = Sum{ j | Q[j]}/n
Compute C = Sum{ j | (Q[j]-P)*(Q[j]-P)'}
diagonalise C and select u to be the eigenvector corresponding to the largest eigenvalue. Note C is symmetric, so diagonalising should not be a problem.