If you are going to use HSV you need to realize that HSV are not points in a three dimensional space but rather the angle, magnitude, and distance-from-top of a cone. To calculate the distance of an HSV value you either need to determine your points in 3d space by transforming.
X = Cos(H)*S*V
Y = Sin(H)*S*V
Z = V
For both points and then taking the Euclidian distance between them:
Sqrt((X0 - X1)*(X0 - X1) + (Y0 - Y1)*(Y0 - Y1) + (Z0 - Z1)*(Z0 - Z1))
At a cost of 2 Cos, 2 Sin, and a square root.
Alternatively you can actually calculate distance a bit more easily if you're so inclined by realizing that when flattened to 2D space you simply have two vectors from the origin, and applying the law of cosign to find the distance in XY space:
C² = A² + B² + 2*A*B*Cos(Theta)
Where A = S*V of the first value, and B = S*V of the second and cosign is the difference theta or H0-H1
Then you factor in Z, to expand the 2D space into 3D space.
A = S0*V0
B = S1*V1
dTheta = H1-H0
dZ = V0-V1
distance = sqrt(dZ*dZ + A*A + B*B + 2*A*B*Cos(dTheta);
Note that because the law of cosigns gives us C² we just plug it right in there with the change in Z. Which costs 1 Cos and 1 Sqrt. HSV is plenty useful, you just need to know what type of color space it's describing. You can't just slap them into a euclidian function and get something coherent out of it.