Video Camera and Stage3D on Mobile Adobe Flash AIR. Augmented-Reality on AIR

Question

Quick Question

How to show webcam rapidly on Adobe AIR application with Stage3D?

---

Detailed question

About

My goal is to create prototype of AR (Augmentation Reality) mobile application. I have chosen Adobe Flash AIR for good 3D graphics support on mobile and AIR apps easy to porting to many mobile platforms (iOS, Android, Blackberry Playbook).

Purpose

I want to show up complex 3D model (so i need to use Stage3D). And underneath a video from Front Camera. As usual AR application.

Here is examples

_{(source: augmentedplanet.com)}

Problem

Stage3D not transparent at all so i can't use StageVideo for the rapid showing of content of Camera, because StageVideo doesn't seen under Stage3D.

So

And only decision i have found - it's to create flat surface with dynamic texture updating.

Here is example of integration of webcamvideo with Starling Framework (Stage3D). But with many ordinary mobile devices we get such a big texture updating (almost as size of screen resolution), that any app will fall down to low fps or even crashes down. What i have done on my Galaxy Note for example. With 320x200 texture size it's has fairly good performance but look ugly at AR app.

So is any brilliant solution for create AR on AIR? Is anybody got same challenge?

Answer 1

This use case is unfortunately not well supported in AIR. Your best bet is really the manual upload. It might help to add votes to feature requests on Adobe forums for transparent Stage3D.

Now for why this feature was low priority: If you are doing AR you are probably already doing CPU work on the video. That means you already read back the camera data for processing either on the CPU or as a Stage3D texture. That's the expensive part, not the uploading a texture back to Stage3D.

In order for this to be useful there would need to be a lot of complicated code paths working together flawlessly. On all supported devices:

• Read back low resolution camera for CPU or GPU AR image processing
• Show passthrough high resolution camera image
• Overlay 3D with blending

This is unfortunately very hard. On many mobile chipsets video/camera, CPU, and 3D are very separate units so it is hard to share data between them without stalling or copying. It can be done very well IF you target specific hardware. I know this does not solve your problem but I hope it explains why this use case does not work well in AIR yet. I think you have those options:

• Go with AIR and readback/upload. It will be very slow on some HW, but it will work reliably.
• Go native. It will be a huge win in the best case, but you need a lot of custom code and testing for every single target.
• Go native on a single very narrow platform. Many very cool AR demos do this. Look at SDKs for AR from GPU vendors. Most of them have one.
• Make the best out of a bad situation: Stick with low res and uploads but add some interesting filters on the video. Once you paid for the upload doing Stage3D stuff with you texture is very cheap.

I hope this helps a bit. While developing Stage3D this exact use case came up every now and then. I still think it is really cool! Maybe this post explained why it never made the top of the list yet.