Android AR in Fragment

Question

I've been trying to get some AR (Augmented Reality) SDK's working in a Fragment. However, I can't seem to get it working.

I've found some code of someone who has got Metaio (AR Framework) working in a fragment.

So I've applied that code to my own project, it is working but the code is not programmed to scan a picture. I want to scan a picture mark with it.

I copied some code to scan a picture mark from a Sample Project of Metaio, but it doesn't work.

Right now it is failing at (Debug logs after that don't get logged):

trackingConfigFile = AssetsManager.getAssetPath(getActivity().getApplicationContext(), "AEDApp/Assets/TrackingData_PictureMarker.xml");

This is my full code:

package com.example.bt6_aedapp;

import android.app.Application;
import android.content.res.Configuration;
import android.hardware.Camera.CameraInfo;
import android.os.Bundle;
import android.support.v4.app.Fragment;
import android.util.Log;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;

import com.metaio.cloud.plugin.MetaioCloudPlugin;
import com.metaio.sdk.MetaioDebug;
import com.metaio.sdk.MetaioSurfaceView;
import com.metaio.sdk.SensorsComponentAndroid;
import com.metaio.sdk.jni.ERENDER_SYSTEM;
import com.metaio.sdk.jni.ESCREEN_ROTATION;
import com.metaio.sdk.jni.IGeometry;
import com.metaio.sdk.jni.IMetaioSDKAndroid;
import com.metaio.sdk.jni.IMetaioSDKCallback;
import com.metaio.sdk.jni.MetaioSDK;
import com.metaio.sdk.jni.TrackingValuesVector;
import com.metaio.sdk.jni.Vector3d;
import com.metaio.tools.Screen;
import com.metaio.tools.SystemInfo;
import com.metaio.tools.io.AssetsManager;

public class fragmentA extends Fragment implements MetaioSurfaceView.Callback {

    private Application mAppContext;

    private ViewGroup mRootLayout;

    String trackingConfigFile;

    private MetaioSDKCallbackHandler mCallback;

    private IGeometry mModel;

    private IMetaioSDKAndroid mMetaioSDK;

    private MetaioSurfaceView mSurfaceView;

    private static boolean mNativeLibsLoaded = false;

    private boolean mRendererInitialized;

    private SensorsComponentAndroid mSensors;

    static {
        mNativeLibsLoaded = IMetaioSDKAndroid.loadNativeLibs();
    }

    @Override
    public void onCreate(Bundle savedInstanceState) {

        MetaioCloudPlugin.startJunaio(null, getActivity().getApplicationContext());

        super.onCreate(savedInstanceState);
        Log.d("LifeCycle", "onCreate");

        mAppContext = getActivity().getApplication();
        mMetaioSDK = null;
        mSurfaceView = null;
        mRendererInitialized = false;
        try {

            mCallback = new MetaioSDKCallbackHandler();

            if (!mNativeLibsLoaded){
                throw new Exception("Unsupported platform, failed to load the native libs");
            }

            // Create sensors component
            mSensors = new SensorsComponentAndroid(mAppContext);

            // Create Unifeye Mobile by passing Activity instance and
            // application signature
            mMetaioSDK = MetaioSDK.CreateMetaioSDKAndroid(getActivity(), getResources().getString(R.string.metaioSDKSignature));
            mMetaioSDK.registerSensorsComponent(mSensors);

        } catch (Throwable e) {
            MetaioDebug.log(Log.ERROR, "ArCameraFragment.onCreate: failed to create or intialize metaio SDK: " + e.getMessage());
            return;
        }
    }

    @Override
    public View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState) {
        Log.d("LifeCycle", "onCreateView");
        View view = inflater.inflate(R.layout.fragment_a, container, false);
        mRootLayout = (ViewGroup)getActivity().findViewById(R.id.pager);
        return view;
    }

    @Override
    public void onStart() {
        super.onStart();
        Log.d("LifeCycle", "onStart");

        if(mMetaioSDK == null){
            return;
        }
        MetaioDebug.log("ArCameraFragment.onStart()");

        try {
            mSurfaceView = null;

            // Start camera
            startCamera();

            // Add Unifeye GL Surface view
            mSurfaceView = new MetaioSurfaceView(mAppContext);
            mSurfaceView.registerCallback(this);
            mSurfaceView.setKeepScreenOn(true);

            MetaioDebug.log("ArCameraFragment.onStart: addContentView(mMetaioSurfaceView)");
            mRootLayout.addView(mSurfaceView, new ViewGroup.LayoutParams(ViewGroup.LayoutParams.WRAP_CONTENT,
                    ViewGroup.LayoutParams.WRAP_CONTENT));
            mSurfaceView.setZOrderMediaOverlay(true);

        } catch (Exception e) {
            MetaioDebug.log(Log.ERROR, "Error creating views: " + e.getMessage());
            MetaioDebug.printStackTrace(Log.ERROR, e);
        }
    }

    @Override
    public void onResume() {
        super.onResume();
        Log.d("LifeCycle", "onResume");

        // make sure to resume the OpenGL surface
        if (mSurfaceView != null) {
            mSurfaceView.onResume();
        }

        if(mMetaioSDK != null){
            mMetaioSDK.resume();
        }
    }

    @Override
    public void onPause() {
        super.onPause();

        Log.d("LifeCycle", "onPause");

        // pause the OpenGL surface
        if (mSurfaceView != null) {
            mSurfaceView.onPause();
        }

        if (mMetaioSDK != null) {
            // Disable the camera
            mMetaioSDK.pause();
        }

    }

    @Override
    public void onStop() {
        super.onStop();

        Log.d("LifeCycle", "onStop");

        if (mMetaioSDK != null) {
            // Disable the camera
            mMetaioSDK.stopCamera();
        }

        if (mSurfaceView != null) {
            mRootLayout.removeView(mSurfaceView);
        }

        System.runFinalization();
        System.gc();
    }

    @Override
    public void onDestroy() {
        super.onDestroy();

        mCallback.delete();
        mCallback = null;

        /*Log.d("LifeCycle", "onDestroy");

        try {
            mRendererInitialized = false;
        } catch (Exception e) {
            MetaioDebug.printStackTrace(Log.ERROR, e);
        }

        MetaioDebug.log("ArCameraFragment.onDestroy");

        if (mMetaioSDK != null) {
            mMetaioSDK.delete();
            mMetaioSDK = null;
        }

        MetaioDebug.log("ArCameraFragment.onDestroy releasing sensors");
        if (mSensors != null) {
            mSensors.registerCallback(null);
            mSensors.release();
            mSensors.delete();
            mSensors = null;
        }

        // Memory.unbindViews(activity.findViewById(android.R.id.content));

        System.runFinalization();
        System.gc();*/
    }

    @Override
    public void onConfigurationChanged(Configuration newConfig) {
        super.onConfigurationChanged(newConfig);
        final ESCREEN_ROTATION rotation = Screen.getRotation(getActivity());
        mMetaioSDK.setScreenRotation(rotation);
        MetaioDebug.log("onConfigurationChanged: " + rotation);
    }

    @Override
    public void onDrawFrame() {     

        if(mMetaioSDK != null) {
            TrackingValuesVector poses = mMetaioSDK.getTrackingValues();

            if(poses.size() != 0) {
                mModel.setCoordinateSystemID(poses.get(0).getCoordinateSystemID());
            }
        }

//      Log.d("LifeCycle", "onDrawFrame");
        /* if (mRendererInitialized) {
            mMetaioSDK.render();
        } */
    }

    @Override
    public void onSurfaceCreated() {
        Log.d("LifeCycle", "onSurfaceCreated");

        try {
            if (!mRendererInitialized) {
                mMetaioSDK.initializeRenderer(mSurfaceView.getWidth(), mSurfaceView.getHeight(), Screen.getRotation(getActivity()),
                        ERENDER_SYSTEM.ERENDER_SYSTEM_OPENGL_ES_2_0);
                mRendererInitialized = true;
            } else {
                MetaioDebug.log("ArCameraFragment.onSurfaceCreated: Reloading textures...");
                mMetaioSDK.reloadTextures();
            }

            MetaioDebug.log("ArCameraFragment.onSurfaceCreated: Registering audio renderer...");
            // mMetaioSDK.registerAudioCallback(mSurfaceView.getAudioRenderer());
            mMetaioSDK.registerCallback(mCallback);

            MetaioDebug.log("ARViewActivity.onSurfaceCreated");
        } catch (Exception e) {
            MetaioDebug.log(Log.ERROR, "ArCameraFragment.onSurfaceCreated: " + e.getMessage());
        }

        mSurfaceView.queueEvent(new Runnable() {
            @Override
            public void run() {
                loadContents();
            }
        });

    }

    private void loadContents() {
        try {       
            trackingConfigFile = AssetsManager.getAssetPath(getActivity().getApplicationContext(), "AEDApp/Assets/TrackingData_PictureMarker.xml");

            boolean result = mMetaioSDK.setTrackingConfiguration(trackingConfigFile);

            Log.d("result", Boolean.toString(result));

            MetaioDebug.log("Tracking data loaded: " + result);

            String aedLogo = AssetsManager.getAssetPath(getActivity().getApplicationContext(), "AEDApp/Assets/metaioman.md2");
            Log.d("aedLogo", "aaa: " + aedLogo);
            if(aedLogo != null) {
                mModel = mMetaioSDK.createGeometry(aedLogo);

                if(mModel != null) {
                    mModel.setScale(new Vector3d(4.0f, 4.0f, 4.0f));
                }
                else {
                    MetaioDebug.log(Log.ERROR, "Error loading geometry: " + aedLogo);
                }               
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    @Override
    public void onSurfaceChanged(int width, int height) {
        Log.d("LifeCycle", "onSurfaceChanged");

        mMetaioSDK.resizeRenderer(width, height);
    }

    @Override
    public void onSurfaceDestroyed() {
        Log.d("LifeCycle", "onSurfaceDestroyed");

        MetaioDebug.log("ArCameraFragment.onSurfaceDestroyed(){");
        mSurfaceView = null;
        // mMetaioSDK.registerAudioCallback(null);
    }

    protected void startCamera() {
        final int cameraIndex = SystemInfo.getCameraIndex(CameraInfo.CAMERA_FACING_BACK);
        if (mMetaioSDK != null) {
            mMetaioSDK.startCamera(cameraIndex, 640, 480);
        }
    }

    final class MetaioSDKCallbackHandler extends IMetaioSDKCallback {
        @Override
        public void onTrackingEvent(final TrackingValuesVector trackingValues) {

            super.onTrackingEvent(trackingValues);
            if(!trackingValues.isEmpty() && trackingValues.get(0).isTrackingState()){
                Log.d("Track", "NOT EMPTY");
            }
        }
    }   
}

I really hope someone can help me with this as I can not figure it out.. :(

EDIT

The Error (e.printStackTrace()) is throwing is:

03-24 20:25:19.068: W/System.err(28062): java.lang.NullPointerException: null string

03-24 20:25:19.068: W/System.err(28062):    at com.metaio.sdk.jni.MetaioSDKJNI.IMetaioSDK_setTrackingConfiguration__SWIG_1(Native Method)

03-24 20:25:19.068: W/System.err(28062):    at com.metaio.sdk.jni.IMetaioSDK.setTrackingConfiguration(IMetaioSDK.java:106)

03-24 20:25:19.068: W/System.err(28062):    at com.example.bt6_aedapp.fragmentA.loadContents(fragmentA.java:278)

03-24 20:25:19.068: W/System.err(28062):    at com.example.bt6_aedapp.fragmentA.access$0(fragmentA.java:274)

03-24 20:25:19.068: W/System.err(28062):    at com.example.bt6_aedapp.fragmentA$1.run(fragmentA.java:268)

03-24 20:25:19.068: W/System.err(28062):    at android.opengl.GLSurfaceView$GLThread.guardedRun(GLSurfaceView.java:1463)

03-24 20:25:19.068: W/System.err(28062):    at android.opengl.GLSurfaceView$GLThread.run(GLSurfaceView.java:1240)

What I want to do with it:

Being able to 'scan' a picture (https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQFqKIurD3QMU0zVeiwEhtm1twLmTCDlnFulfCwDkxTA1_XQjIQ) and detect the image in the app. The image is referenced in the app in the Assets folder of the project, and I've made a xml file where the marker for it is defined as is stated on the Metaio website. After detecting I'm going to do some database stuff, but for now I need to get the detecting part working.

EDIT If anyone knows how i can make another AR Framework in Fragments I would love tot know.

Well, for Metaio i followed a webinar about Getting Started with Metaio and setting it up in a new project. However, there they use activities, I really don't know how to make it so that it works on a Fragment. — Kevin Sleegers, Mar 24 '14 at 15:34
What does “it is failing” mean? It throws an exception? Tell us the exception. It doesn't do what you want? What do you want? What does it do? — Dour High Arch, Mar 24 '14 at 19:21
If anyone has a solution on how to use other AR Frameworks like Vuforia in a Android Fragment, please let me know.. This is really freaking me out.. It's been 14 hours since I started with this and I really don't know what to do. Nothing seems to work, except for the example projects. But I can't seem to get them to work in my own application .. :/ — Kevin Sleegers, Mar 24 '14 at 21:40

score 1 · Answer 1 · answered May 29 '14 at 11:54

I don't know pretty much anything about fragments but as for the null string I think that happens because you haven't extracted the assets.

In this video http://youtu.be/KVtCi-WwmFU?t=30m29s it's explained. Basically what you have to do is add this code

private class AssetsExtracter extends AsyncTask<Integer, Integer, Boolean>{
    @Override
    protected Boolean doInBackground(Integer... params){
        try 
        {
            AssetsManager.extractAllAssets(getApplicationContext(), BuildConfig.DEBUG);
        }catch (IOException e){
            MetaioDebug.printStackTrace(Log.ERROR, e);
            return false;
        }
        return true;
    }
}

to your activity (or in this case, I guess, your fragment). Then you have to add a field of this class like

private AssetsExtracter mTask;

and inside the onCreate() method you put

mTask = new AssetsExtracter();
mTask.execute(0);

After that your assets should be avaliable from AssetsManager.getAssetPath(..) and it shouldn't return a null string anymore.

Still getting Null after this – Saurabh Nov 12 '14 at 12:16 — Saurabh, Nov 12 '14 at 12:16

Android AR in Fragment

1 Answers1