2

Is there a way to apply a full screen shader to a RealityKit scene (Swift/iOS)?

I know I can do this with SCNTechnique in SceneKit/ARSCNView, but I'm dealing with an ARView.

  • Is there a way in RealityKit one can apply a vertex/fragment shader?
  • or a way to get the rendered scene as a texture?

With an ARSCNView I can apply a shader to run over the output of a scene, just like: https://medium.com/ar-tips-and-tricks/arkit-theory-an-introduction-to-scntechniques-710e024bc91e

But with RealityKit, I'm dealing with an ARView which doesn't have a SCNTechnique property

Thanks :-)

Chris
  • 2,739
  • 4
  • 29
  • 57
  • Hi @Chris! What `full screen shader` do you mean exactly? Environment shader? – Andy Jazz Apr 23 '20 at 12:42
  • Hi Andy, Anything at the minute, I've been trawling the docs, but can't seem to find a reference to apply ANY shader – Chris Apr 23 '20 at 13:02
  • Sorry))) Shader to what? To environment or to a model? What do you mean saying `full screen shader`? – Andy Jazz Apr 23 '20 at 13:12
  • In a ARSCNView I can apply a SCNTechnique over the full screen, as if I'm dealing with a texture. but in RealityKit, I'm dealing with an ARView – Chris Apr 23 '20 at 13:53
  • Ok, I've got it. You need `render passes`, also known as `AOV`s. https://developer.apple.com/documentation/scenekit/scntechnique They are not shaders. They are Render passes like `Point Position` pass, `Normals` pass, `ZDepth` channel pass, `UV` pass, etc. But shaders are `Phong`, `Lambert`, `Blinn`, etc. – Andy Jazz Apr 23 '20 at 13:59
  • 1
    Apologies for using the wrong terminology, is there something similar in ARView? – Chris Apr 23 '20 at 14:09
  • Alas @Chris, at the moment No. – Andy Jazz Apr 23 '20 at 14:15

2 Answers2

3

iOS RealityKit

In RealityKit for iOS you can apply background color/image, HDR for image based lighting (IBL). You can apply environmental image via skybox(:_) type method.

func skybox(_ resource: EnvironmentResource) -> ARView.Environment.Background

This static method uses EnvironmentResource class that contains background and lighting data for your scene. You can read about it here and in my story on Medium.

Also, in RealityKit for iOS you are capable of applying a vertex/fragment shader (via CustomMaterial) and get the rendered scene as a texture (like you could do in SceneKit+Metal).


visionOS RealityKit

Read this post, showing you how to enable an environment lighting in visionOS app.

Andy Jazz
  • 49,178
  • 17
  • 136
  • 220
  • 1
    do we still have to make ```folderName.skybox/file.hdr``` I'm adding via ```let skyboxName = "rooitou_park_1k.hdr" // The .exr or .hdr file``` ```let skyboxResource = try! EnvironmentResource.load(named: skyboxName)``` ```environment.lighting.resource = skyboxResource``` ```environment.background = .skybox(skyboxResource)``` All I got is black background and generic lighting on object :- ( – Dariusz Jun 26 '21 at 09:45
  • You can do it Apple's way or your own way – it's up to you. The main thing – it must work. Although, I want to point out that this option is buggy. – Andy Jazz Jun 26 '21 at 09:55
  • Yeh, just lost my bananas with it. Ended up the name... have to use name, and not name.ext... ehh happy time. thanks you for quick reply! Is there a way to rotate the ibl? and undo ibl/go back to automatic? – Dariusz Jun 26 '21 at 10:00
  • I cannot see the way to rotate IBL (it's a problem for many 3D engines). You must prepare your HDRI file appropriately. – Andy Jazz Jun 26 '21 at 10:06
  • My example for `three.js `https://stackoverflow.com/questions/64454022/is-it-possible-to-rotate-a-cube-environment-map-180-degrees-around-y-axis/64460378#64460378 – Andy Jazz Jun 26 '21 at 10:08
  • Hmm pretty odd... I imagine its just a matter of matrix4x4 * hdr to give it different transform/use for scene lighting. Pretty much any engine I deal with allow for rotation/scale of hdr/map coordinates. This is gonna be difficult task then as I wanted to match hdr rotation to sun position to simulate better "environment" but preserving more or less similar lighting direction – Dariusz Jun 26 '21 at 10:11
  • Probably, it'll be implemented in RealityKit 3.0. – Andy Jazz Jun 26 '21 at 10:12
0

There's a whole pipeline now for post processing, with CIFilters, SpriteKit or Metal Performance shaders. See this talk: https://developer.apple.com/videos/play/wwdc2021/10075/

Niklas
  • 1,322
  • 14
  • 11