3

I am trying to build an app on iOS in Swift, SwiftUi and using ARKit and RealityKit. I want the app to:

  1. detect a soccer ball
  2. detect a soccer goal
  3. track the trajectory of the ball when shot at goal
  4. detect when/if the ball hits/passes the goal

To detect the ball and goal using ARKit is working fine. However, to keep tracking the ball when in movement is very unreliable. And when its moving fast, ARKit fails to detect the ball at all.

I have tried to implement Apple Visions VNDetectTrajectoriesRequest using the following tutorial,https://developer.apple.com/documentation/vision/identifying_trajectories_in_video, which works as long as I am NOT using ARKit. DetectTrajectory needs a CMSampleBuffer, and ARKit only returns a PixelBuffer and I wasn't able to solve this issue and get it to work smoothly (I tried converting the pixelbuffer to CMSampleBuffer, but that resulted in the app being SO slow it was useless in realtime). And even if I get this to work it wouldn't solve my issue regarding detecting when the soccer hits the goal.. For that I believe that using depth/ARkit is necessary..

Does anyone have a solution for tracking A FAST MOVING OBJECT USING ARKIT?

I am close to giving up, but I have found this guy on Youtube that is detecting a balls trajectory using ARKit, https://www.youtube.com/watch?v=B4yfp1UmM5s. I have written to him but received no reply this far. Have been trying to solve this for over a week but I can't find a solution. Please internet, help me!

Andy Jazz
  • 49,178
  • 17
  • 136
  • 220
sofiarotten
  • 31
  • 1
  • Hi, Did you check this sample code here? https://developer.apple.com/documentation/arkit/content_anchors/recognizing_and_labeling_arbitrary_objects This sample code shows how to integrate ARKit and Vision. The key to the performance issue should be. 1. Retain the CVPixelBuffer for VNImageRequestHandler(cvPixelBuffer: currentBuffer!, orientation: orientation). 2. Perform the classification request in another thread. ex: visionQueue = DispatchQueue(label:. 3. Release the buffer for next time usage. (important). I hope this sample code can help on this exciting topic. – Nathan Chang Jun 10 '22 at 23:07
  • Here is a reference note in the [link](https://developer.apple.com/documentation/arkit/content_anchors/recognizing_and_labeling_arbitrary_objects). ``` Important Limit your processing to one buffer at a time for performance. The camera recycles a finite pool of pixel buffers, so retaining too many buffers for processing could starve the camera and shut down the capture session. Passing multiple buffers to Vision for processing would slow down processing of each image, adding latency and reducing the amount of CPU and GPU overhead for rendering AR visualizations. ``` – Nathan Chang Jun 10 '22 at 23:08

1 Answers1

0

I think that such a mission is impossible for iOS device in 2022.

Firstly, let's assume the average speed of a soccer ball is 12 m/s, and ARKit and Vision track it at 60 fps. Any object moving at 12 m/s is difficult to qualitatively track at that frame rate, it's obvious. Even MoCap systems use at least 120 fps for tracking of much slower movements.

Secondly, in 5 seconds the ball approximately covers a distance of 60 meters. This is a fairly large distance (for detection/recognition) at which such a small object as a soccer ball will be unrecognizable, especially since it also spins during flight.

Andy Jazz
  • 49,178
  • 17
  • 136
  • 220
  • 2
    Thank you so much for your reply. I agree it feels like an impossible mission at this point. The only thing that makes me question if I've missed anything is that YouTube video I linked in my post. That guy is obviously tracking a moving object, and he is using ARKit.. https://www.youtube.com/watch?v=B4yfp1UmM5s – sofiarotten Apr 15 '22 at 10:19
  • @sofiarotten, the average speed of soccer ball is 129 km/h (to google it), guy's hand-ball moves with a speed of 25...50 mph (40...80 km/h). And the distance at which the hand-ball moves away is about 10 meters, not 50 meters. However, no one forbids you to perform an experiment in a football stadium. Just try! – Andy Jazz Apr 15 '22 at 10:46