Checking if vector passes through vertices

Question

I have been struggling with this problem for over a month, so I really need your help.

To further elaborate on the question :

The question is whether a vector called 'direction' that starts at a vertex called 'start' passes through the 'taget'.

You need to confirm the direction and distance.

I decided that using the dot product was impossible because I went through enough debugging.

1. The result is good when calculated directly, but why is the result different when executed in the shader?

2. The same thickness should be printed depending on the distance, but why does the thin line appear when the distance is far?

3. Do you have any good ideas even if it's not the way I use the rotation matrix?

These are three questions. First of all, my situation is

1. drawing fSQ.

2. I want to check whether the direction of start passes through the target.

3. Compute in the pixel shader.

4. 1 is one pixel

5. The screen size is 1920*1080

    bool intersect(float2 target, float2 direction, float2 start) {
   
    bool intersecting = false;
    static const float thresholdX = 0.5 / SCREENWIDTH;
    static const float thresholdY = 0.5 / SCREENHEIGHT;
   
    if (direction.x == 0 && direction.y == 0);
   
    else {
        float2 startToTarget = target - start;
        float changedTargetPositionX = startToTarget.x * direction.x + startToTarget.y * direction.y;
        float changedTargetPositionY = startToTarget.x * (-direction.y) + startToTarget.y * direction.x;
        float rangeOfX = (direction.x * direction.x) + (direction.y * direction.y);
   
        if (changedTargetPositionX <= rangeOfX + thresholdX && changedTargetPositionX >= -thresholdX &&
            changedTargetPositionY <= thresholdY && changedTargetPositionY >= -thresholdY) {
            intersecting = true;
        }
    }
   
    return intersecting;
   

We use a rotation matrix to rotate a vector and then check the difference between the two vectors, which works in most cases, but fails for very small pixels.

For example

start = (15,0) direction= (10,0) taget = (10,0)

In this case, the intersect function should return false, but it returns true.

But if the pixel difference is bigger then it works fine. and

#define MAX = 5;
float2 points[MAX*MAX];
for (float fi = 1; fi < MAX; fi++)
    for (float fj = 1; fj < MAX; fj++)
        points[(int)(fi * MAX + fj)] = float2(fi / MAX , fj / MAX);

for(uint ni=0; ni < MAX*MAX;ni++)
    for(uint nj=3; nj < MAX*MAX; nj++)
        if (intersect(uv,  points[nj]- points[ni], points[ni])) {

            color = float4(1, 0, 0, 1);
            return color;
        }
return float4(0, 0, 0, 1);

When debugging like this, the line becomes thinner depending on the distance. All the lines should have the same thickness, but I don't know why.

This is the result of running the debugging code:

We look forward to your reply. thank you