1

I have made project with ARKit that uses metal shaders to perform a mirroring effect, and this is assigned by using sceneView.technique. I am trying to figure out if it would be possible to transition the project over to using RealityKit instead, but cannot find any similar trait of the ARView used by RealityKit. Is there anything in RealityKit similar to sceneView.technique that would allow me to simply assign the metal shaders as I do right now, or would I have to rework how the metal shaders are interacting with the view?

My current implementation is based on this post.

Charlie
  • 136
  • 1
  • 8

2 Answers2

1

About ARView mirroring

The simplest way to mirror ARView in RealityKit 2.0 is to use a CGAffineTransform 3x3 matrix.

 ┌             ┐
 |  a   b   0  |
 |  c   d   0  |
 |  tx  ty  1  |
 └             ┘

We need to reflect a scene (including video) horizontally, so we have to use -1 multiplier for xScale:

 ┌             ┐
 | -1   0   0  |
 |  0   1   0  |
 |  0   0   1  |
 └             ┘

Here's the code:

arView.transform = .init(a: -1, b: 0, c: 0, d: 1, tx: 0, ty: 0)


About Metal shaders

In RealityKit 2.0 only CustomMaterial allows you implement Metal's vertex and fragment shaders. So, you can use a vertex shader for many interesting effects, including flipping a model across X axis.

.swift file:

import Metal
import RealityKit

let device = MTLCreateSystemDefaultDevice()
    
guard let defaultLibrary = device!.makeDefaultLibrary()
else { return }
    
let shader = CustomMaterial.SurfaceShader(named: "basicShader",
                                             in: defaultLibrary)
    
let modifier = CustomMaterial.GeometryModifier(named: "basicModifier",
                                                  in: defaultLibrary)
    
box.model?.materials[0] = try! CustomMaterial(surfaceShader: shader, 
                                           geometryModifier: modifier, 
                                              lightingModel: .lit)

.metal file:

#include <metal_stdlib>
#include <RealityKit/RealityKit.h>

using namespace metal;
using namespace realitykit;

[[visible]]
void basicShader(realitykit::surface_parameters shader)
{
    surface::surface_properties sf = shader.surface();
    
    sf.set_base_color( half3(1.0f, 0.7f ,0.0f) );
};

[[visible]]
void basicModifier(realitykit::geometry_parameters modifier)
{
    float3 pose = modifier.geometry().model_position();

    // your code...
};

However, a counterpart of SCNTechnique for RealityKit view isn't "baked" yet...


Andy Jazz
  • 49,178
  • 17
  • 136
  • 220
  • 1
    Just to clarify, is this for for mirroring the camera feed itself or for making a mirrored material that would be applied to virtual content? I am trying to mirror the whole camera feed, apologies if that wasn't clear. – Charlie Feb 16 '22 at 02:18
  • I tried putting in `arView.transform = .init(a: -1, b: 0, c: 0, d: 1, tx: 0, ty: 0)` immediately after the initialization of the ARView via `var arView = ARView(frame: .zero)` and got a whole slew of errors. It's saying "consecutive declarations on a line must be separated by ';'", and appears to think that the `arView.transform` line is trying to declare a function. Is there a different initializer I need to use for the CGAffineTransform, or a different way to apply it to the `arView`? – Charlie Feb 17 '22 at 19:12
  • 1
    It 100% works fine – I applied it many times. – Andy Jazz Feb 17 '22 at 19:16
  • 1
    Realized I just needed to move the transform statement into my makeUIView function instead of having it in the ARViewContainer struct. Thanks so much for all your help! – Charlie Feb 17 '22 at 19:47
0

You can use RealityKit 2's PostProcessing API which should make this desired effect pretty easy to implement.

For example you could apply a Core Image Filter or Metal kernel functions to your rendered scene. Somewhat similar to SCNTechnique.

https://developer.apple.com/documentation/realitykit/implementing_special_rendering_effects_with_realitykit_postprocessing?changes=l_6&language=objc

arthurschiller
  • 316
  • 2
  • 9