Apple Vision image recognition

Question

As many other developers, I have plunged myself into Apple's new ARKit technology. It's great. For a specific project however, I would like to be able to recognise (real-life) images in the scene, to either project something on it (just like Vuforia does with its target images), or to use it to trigger an event in my application.

In my research on how to accomplish this, I stumbled upon the Vision and CoreML frameworks by Apple. This seems promising, although I have not yet been able to wrap my head around it.

As I understand it, I should be able to do exactly what I want by finding rectangles using the Vision framework and feeding those into a CoreML model that simply compares it to the target images that I predefined within the model. It should then be able to spit out which target image it found.

Although this sounds good in my head, I have not yet found a way to do this. How would I go about creating a model like that, and is it even possible at all?

Funny, I was wondering how exactly that works just yesterday. Thanks for asking! — LinusGeffarth, Sep 11 '17 at 09:12

score 3 · Answer 1 · answered Sep 11 '17 at 15:10

3

I found this project on Github some weeks ago: AR Kit Rectangle Detection

I think that is exactly what you are looking for...

answered Sep 11 '17 at 15:10

Nico S.

3,056
1
30
64

Yes, this is very nice, thank you. However, this is just the basis of what I was looking for, as this application is able to detect a rectangle, but does not distinguish between different images on that rectangle. I think that should be possible using CoreML, however I'm not certain on how I would go about doing that. – Marc Van Deuren Sep 12 '17 at 08:03
I'm looking for exactly the same. Did you already find a solution? I was thinking about training a model. After ARKit with CoreML recognises the image find the rectangle and you are having what you want. But training an CoreML model for just one image is too much I think... – KNV Oct 31 '17 at 16:42
Sadly for you, I was thinking just the same thing, and have not yet found an answer. For the project I'm currently working on, I tried combining ARToolkit and ARKit (the former just for the image recognition). This is certainly possible through Unity, though I have not yet finished the process. If you wanted to do it natively, however, I'm not sure that you could. I do know Vuforia plans to release Vuforia Fusion sometime in the beginning of next year, which will combine its image recognition capabilities with ARKit and/or even ARCore. – Marc Van Deuren Nov 02 '17 at 08:00
I'm trying to do the same thing. Was thinking of processing the images using AWS Rekognition, but I still need a way to know locally if a face has been processed or not. – davidmerrick Dec 31 '17 at 20:14

score 0 · Accepted Answer · answered Feb 06 '18 at 11:15

As of ARKit 1.5 (coming with IOS 11.3 in the spring of 2018), a feature seems to be implemented directly on top of ARKit that solves this problem.

ARKit will fully support image recognition. Upon recognition of an image, the 3d coordinates can be retrieved as an anchor, and therefore content can be placed onto them.

score 0 · Answer 3 · answered Mar 29 '20 at 10:32

Vision's ability to detect images was implemented in ARKit beginning from iOS 11.3+, so since then ARKit has ARImageAnchor subclass that extends ARAnchor parent class and conforms to ARTrackable protocol.

// Classes hierarchy and Protocol conformance...

ObjectiveC.NSObject: NSObjectProtocol
        ↳ ARKit.ARAnchor: ARAnchorCopying
                ↳ ARKit.ARImageAnchor: ARTrackable

ARWorldTrackingConfiguration class has a detectionImages instance property that is actually a set of images that ARKit attempts to detect in the user's environment.

open var detectionImages: Set<ARReferenceImage>!

And ARImageTrackingConfiguration class has a trackingImages instance property that is a set as well, it serves the same purpose – ARKit attempts to detect and track it in the user's environment.

open var trackingImages: Set<ARReferenceImage>

So, having a right configuration and an ability to automatically get ARImageAnchor in a ARSession, you can tether any geometry to that anchor.

P.S. If you want to know how to implement image detection feature in your ARKit app please look at this post.

Apple Vision image recognition

3 Answers3