How does Google launch the AR Animals ARCore app from a browser link?

Question

TL;DR;

Google's new AR animals appear as if they are AR on the web (like AR.js or 8thWall), but are actually native ARCore applications. The sleight of hand is that a link in a browser (actually a button with a very obfuscated event handler chain) launches the ARCore app itself without any prompt for the user to install anything. How does Google do this? I want to do it too.

Detail

Recently Google introduced a search feature that allows users to view animals in AR by clicking a link from a search info card.

This is true plane-sensing, SLAM-tracking AR, no markers required. Viewing examples on my own phone (Pixel 2 running Android 9), the location and angle fidelity is impressive. I can move the phone around, walk around, and the tiger's foot (for example) stays anchored within a few inches. Likewise the AR objects have excellent visual stability, avoiding the shake that plagues most of the marker-based AR.js apps I've seen, or the mediocre object anchoring I see in 8th Wall's non-marker examples.

As a developer working on AR delivered over the web, I'm mighty curious how they pulled off such a high-quality result. Do they have far superior proprietary tech they're not sharing with the THREE.js / ARKit / AR.js open source projects?

No. Debugging into the the tiger example using USB remote inspection of my phone from my laptop, plus some circumstantial clues, I've concluded that they look so good because they are not web AR, but native AR using ARCore. Somehow Google causes the user's phone to load a native ARCore application (system logs make it appear is ARCore itself is being launched as an application) without any install prompt. Attempting to switch to another app causes the AR app to automatically close, making it difficult to debug or inspect it.

Google's own ARCore docs generally imply that you need to publish an app to the Play store to actually get ARCore enabled software into the hands of users. So, does anyone know how Google does what they're doing here?

All images are my original work

Info card link:

Prompt to grant app permissions to ARCore:

ARCore was just used:

Answer 1

https://developers.google.com/ar/develop/java/scene-viewer

Scene Viewer is an immersive viewer that enables AR experiences from your website. It lets Android mobile device users easily place, view, and interact with web-hosted 3D models in their environment.

To do this, all a user needs is an Android device that has ARCore 1.9 or later. Most Android browsers are supported, and no programmatic integration with a browser is necessary -- only properly-formatted links on a web page.

<model-viewer ar alt="A 3D model of an astronaut." src="Astronaut.gltf"></model-viewer>

Thanks for posting this question. :) I was waiting for this feature but I didn't know when it came out.

Answer 2

It works by using Android's intent URLs.

intent://arvr.google.com/scene-viewer/1.2?file=https://storage.googleapis.com/ar-answers-in-search-models/static/GiantPanda/model.glb&title=Giant%20panda&referrer=google.com:ANIMALS:kp_carousel&sound=https://storage.googleapis.com/ar-answers-in-search-models/static/GiantPanda/Bear_Panda_Giant_Unisex_Adult.ogg&card_content=https://arvr.google.com/searchar/infocard?data%3DCg0vZy8xMWo1ZjVkbms0EAE%26hl%3Den-US&share_text=See%20a%20life-sized%20animal!%20Search%20%22Giant%20panda%22%20on%20Google%20and%20tap%20%22View%20in%203D.%22%20https://www.google.com/search?q%3DGiant%2Bpanda%26hl%3Den-US%26stick%3DH4sIAAAAAAAAAFPi0M_VNzBOyjJ8xGjMLfDyxz1hKa1Ja05eY1Th4grOyC93zSvJLKkUEuNig7J4pLi44Jp4FrFyu2cm5pUoFCTmpSQCADr-Ul5OAAAA#Intent;package=com.google.android.googlequicksearchbox;scheme=https;S.browser_fallback_url=https://arvr.google.com/scene-viewer?file=https://storage.googleapis.com/ar-answers-in-search-models/static/GiantPanda/model.glb&title=Giant%20panda&referrer=google.com:ANIMALS:kp_carousel&sound=https://storage.googleapis.com/ar-answers-in-search-models/static/GiantPanda/Bear_Panda_Giant_Unisex_Adult.ogg;end;

Skip to the end for a generator.

Let's break down the URL format.

• intent:// Ok, this is an Android Intent URL

• arvr.google.com/scene-viewer/1.2?[parameters] This is the URL that will be passed to the ARCore app. The parameters apper to work as follows:

  • card_content: This is the link to a webpage to show on the info sheet at the bottom of the AR view.
  • file: This is the 3D model file, in glb format.
  • referrer: Not sure what this is for, tracking probably.
  • share_text: This sets the text to share when you click the share button.
  • sound: An audio file to play, in ogg format.
  • title: The model's name, shown in the title of the info sheet at the bottom.

• #Intent; Alright now we are getting to the part where we tell Android where to open the link.

  • package=com.google.android.googlequicksearchbox; Interesting. This opens in the Google app, not the ARCore app. Google has a habit of sticking random things in the Google app, and I guess this is one of them.
  • scheme=https; This means that the URL at the beginning should be passed to Google as an https:// URL.
  • S.browser_fallback_url=https://arvr.google.com/scene-viewer?[parameters] This URL will be opened if the user doesn't have the Google app.
    • file: See above.
    • title: See above.
    • sound: See above.
    • referrer: See above.

• ;end; The end of the URL is here.

urlform.oninput=()=>{
out.textContent=`intent://arvr.google.com/scene-viewer/1.2?file=${encodeURIComponent(file.value)}&title=${encodeURIComponent(title.value)}&referrer=&sound=${encodeURIComponent(sound.value)}&card_content=${encodeURIComponent(card.value)}&share_text=${encodeURIComponent(share.value)}#Intent;package=com.google.android.googlequicksearchbox;scheme=https;S.browser_fallback_url=https://arvr.google.com/scene-viewer?file=${encodeURIComponent(file.value)}&title=${encodeURIComponent(title.value)}&referrer=&sound=${encodeURIComponent(sound.value)};end;`
}

#urlform{
display:grid;
grid-gap:9px;
}

<form id="urlform">
<input type="text" placeholder="URL of model file (GLB)" id="file">
<input type="text" placeholder="Model name" id="title">
<input type="text" placeholder="URL of sound file (OGG)" id="sound">
<input type="text" placeholder="URL of card webpage (HTML)" id="card">
<textarea placeholder="Share text" id="share"></textarea>
</form>
<div id="out"></div>