2

Context:

I am trying to create a Unity app for a mobile device that can recognize big objects like cars through the device's physical camera.

I was looking at Wikitude and Vuforia but they both only seem to work with small objects that can fit on a table top. Unless I am missing something?

I was also looking at Open CV and ARCore, but they only seem to work with rectangular "markers". Cars for example, is not rectangular and although the shape of the car will always be the same from the same angle, the background and the objects seen behind the car through the windows always differ causing the AR library to try to match pixels that are not part of the car.

Question:

How can one create a Unity app that can recognize a big object or a non-rectangular image through a mobile's physical camera?

Jasper Citi
  • 1,673
  • 1
  • 23
  • 31
  • Dude If you find this out please let me know ^^ Specifically to cars: you might anyway get some issues with reflective and refractive materials. The closest I came so far was [VisionLib](https://visionlib.com) it uses Edge detection. it is a research project by [Fraunhofer](https://www.igd.fraunhofer.de/en/projects/visionlib-augmented-reality-tracking-library-industries). Unfortunately I didn't have the time to actually test it yet but it looks very promising (not for comercial usage) They are quite responsive and happy over feedback – derHugo Apr 16 '19 at 14:17
  • If you have enough data, use Deep learning models – Shubhankar Mohan Apr 16 '19 at 14:35

1 Answers1

2

Image coming from the camera does not contain scale of the object in any way - its just an array from pixels, as captured from the light detecting device (typicaly a Cmos sensor). The appliaction has no way of knowing if the object is big or small (at least until we get into depth cameras), so the methodology in recognising an image is exactly the same for big and for small objects.

As far as root of your question goes : recognizing an object based on the image is a very tricky subject, that only gets managable if we only teach the algorithm to recognise a marker or a QRCode - those are designed specifically to be easily recognizable, that's why ther's more libraries doing that sort of thing.

Vuforia tries to recognize all objects, which is amazing, and will not care about the scale of the object, so you can try to use it, this is pretty much it as far as easy solutions go.

However, this is likely to only work in the same lighting condititions, and will fail if the weather changes etc, if the car is dirty, wet and what not.

The general case, as in aproaching a human-like ability to recognize objects - not yet possible, humans train their neural networks for many years before they can recognize cars.

For more information please read this really informative thread on image recognition (in this case a Coca-Cola can which is far simpler than a general 'car')

Image Processing: Algorithm Improvement for 'Coca-Cola Can' Recognition

Here's an image showing that larger stuff looks the same for the camera

enter image description here

zambari
  • 4,797
  • 1
  • 12
  • 22
  • There is, technically, a discernable difference when scale is involved...because of perspective. A phone's camera is a very wide field of view lens, so larger farther away objects will have their "parts" overlap or curve away differently than a small close object (assuming a perfect replica that differs only on scale). This is why trying to take pictures of landscapes with your phone never turns out the way you expect. – Draco18s no longer trusts SE Apr 16 '19 at 17:34
  • @Draco18s this is not true as far asI know. you dcan't measure how far a photon has traveled since it's last bounce, so it doesn't matter, as long as they hit the same place. If you make something twice as big, and increase the the distance to match, you will get exactly the same image. this is quite easy to prove within unity itself. for the object of a fixed size, yes, distance will matter (edited the answer) – zambari Apr 16 '19 at 18:01
  • 1
    @zambari I understand what you are saying about the size of the objects from a camera's perspective, however Vuforia clearly answers the question "Can I recognize large objects, like a car or a building? Object Recognition is optimized for objects that can fit on a tabletop and are found indoors." (on https://library.vuforia.com/content/vuforia-library/en/articles/Training/Object-Recognition.html) I guess it is probably because the library needs to map the environment from different angles and a large object would make it very difficult as you need to move around that object. – Jasper Citi Apr 16 '19 at 18:36
  • moving around the object is not a major problem, changing lighting conditions are, also I imagine they need a certain amount of parallax to pick up geometry, which is harder to achieve for a building than for a teapot. thanks for pointing out the reference, I was't aware of this recommendation – zambari Apr 16 '19 at 18:49
  • @zambari Agreed, changing lightning conditions could be a problem especially outdoors. What if one forces a very high contrast and/or try to normalize the exposure on the camera input (is that possible in Vuforia)? One should then also create a custom 3D model with similar contrast as a reference. Hopefully the position of the wheels, window frame, grill, bumper etc. with a possible contrasting background will be enough to recognize a car as I guess the shape will kind of get lost on high contrast. – Jasper Citi Apr 17 '19 at 05:44