AKKit: How to select a group of 3D points from a 2D frame?

Question

so the quest is this, I got an ARPointCloud with a bunch of 3d points and I'd like to select them based on a 2d frame from the perspective of the camera / screen.

I was thinking about converting the 2d frame to a 3d frustum and check if the points where the 3d frustum box, not sure if this is the ideal method, and not even sure how to do that.

Would anyone know how to do this or have a better method of achieving this?

If you’re using ARKit with SceneKit, I’d recommend looking into the `projectPoint` function to convert the whole set of points to 2D screen space and check for containment in the rest that way. Constructing a frustum based on arbitrary side planes (as opposed to a viewing camera) tends to be more cumbersome. — rickster, Oct 31 '17 at 15:47
@rickster That's a great idea! Thanks. I did solve it in a different way, by creating a frustum and splitting it up into two tetrahedrons, then I found this script to check if a point is inside one of the tetrahedrons: https://stackoverflow.com/a/25180294/4586652 Tho it's slow, so I'll test your idea! — Heestand XYZ, Nov 01 '17 at 12:54

wcochran · Accepted Answer · 2022-12-20T19:12:43.197

Given the size of the ARKit frame W x H and the camera intrinsics we can create planes for the view frustum sides.

For example using C++ / Eigen we can construct our four planes (which pass through the origin) as

    std::vector<Eigen::Vector3d> _frustumPlanes;
    frustumPlanes.emplace_back(Eigen::Vector3d( fx, 0,   cx - W));
    frustumPlanes.emplace_back(Eigen::Vector3d(-fx, 0,  -cx));
    frustumPlanes.emplace_back(Eigen::Vector3d( 0,  fy,  cy - H));
    frustumPlanes.emplace_back(Eigen::Vector3d( 0, -fy, -cy));

We can then clip a 3D point by checking its position against the z < 0 half-space and the four sides of the frustum:

    auto pointIsVisible = [&](const Eigen::Vector3d& P) -> bool {
        if (P.z() >= 0) return false;  // behind camera
        for (auto&& N : frustumPlanes) {
            if (P.dot(N) < 0)
                return false;  // outside frustum plane
        }
        return true;
    };

Note that it is best to perform this clipping in 3D (before the projection) since points behind or near the camera or points far outside the frustum can have unstable projection values (u,v).

AKKit: How to select a group of 3D points from a 2D frame?

1 Answers1