Is refraction in SceneKit possible?

Question

Is it possible to make a shape that passes through light so you can see through it with the light bent due to refraction? Like a lens or a glass (or water)?

Answer 1

To achieve refraction with SceneKit, you will need a SCNProgram. The built in shaders cannot do anything with refraction.

Based on the answer in this article (Which are the right Matrix Values to use in a metal shader passed by a SCNProgram to get a correct chrome like reflection) a refraction effect with SceneKit can be achieved like this. (This example is based on ARKit)

You need:

• Swift
• SceneKit / ARKit
• SkyBox (we need something to Refract and Reflect)
• SCNProgram
• Metal Shaders
• Physical Device (Recommended)

Make a fresh ARKit (SceneKit) project, create or load a sphere-like geometry object and put it in space. (SCNSphere will be okay)

Implement the skybox. Make sure your skybox composes of 6 individual images (Cube Map) - do not use 2:1 sphere maps, they seem not to work with the sampler in the metal shader. Here is a good link for skyboxes: (https://www.humus.name)

Make a the six required UIImages that holds the individual pictures for the skybox like so:

let skybox1 = UIImage.init(named: "art.scnassets/subdir/image-PX.png")
let skybox2 = UIImage.init(named: "art.scnassets/subdir/image-NX.png")
let skybox3 = UIImage.init(named: "art.scnassets/subdir/image-PY.png")
let skybox4 = UIImage.init(named: "art.scnassets/subdir/image-NY.png")
let skybox5 = UIImage.init(named: "art.scnassets/subdir/image-PZ.png")
let skybox6 = UIImage.init(named: "art.scnassets/subdir/image-NZ.png")

Images must be squares, and should be a power of 2 (for optimal mipmapping purposes). 512x512 will be good, 1024x1024 requires already a lot of memory

Make a SCNMaterialProperty that holds the array of individual images for the skybox like so:

// Cube-Map Structure:
//      PY
//  NX  PZ  PX  NZ
//      NY

// Array Order:
// PX, NX, PY, NY, PZ, NZ

let envMapSkyboxMaterialProperty = SCNMaterialProperty(contents: [skybox1!,skybox2!,skybox3!,skybox4!,skybox5!,skybox6!])

(P = positive, N = negative)

Then set the Skybox like so: (We need this for the reflective, refractive background and lighting of the Scene*)

myScene.background.contents = envMapSkyboxMaterialProperty?.contents

Set the Lighting environment as well**.

myScene.lightingEnvironment.contents = envMapSkyboxMaterialProperty?.contents

Assuming, that by now you can put your geometry object in space with a default material - we are now ready to line up the SCNProgram, with the special Metal Shaders for light refraction.

Make the SCNProgram and configure it like so:

let sceneProgramRefract = SCNProgram()
sceneProgramRefract.vertexFunctionName   = "myVertexRefract" // (myVertexRefract is the Keyword used in the shader)
sceneProgramRefract.fragmentFunctionName = "myFragmentRefract" // (myFragmentRefract is the Keyword used in the shader)

On your target Geometry-Node’s material attach the SCNProgram like so:

firstMaterial.program = sceneProgramRefract // doing this will replace the entire built-in SceneKit shaders for that object.
firstMaterial.setValue(envMapSkyboxMaterialProperty, forKey: "cubeTexture") // (cubeTexture is the Keyword used in the shader to access the Skybox)

Add a new Metal file to your project and call it “shaders.metal”

Replace anything in the Metal file with this:

// Default Metal Header for SCNProgram
#include <metal_stdlib>
using namespace metal;
#include <SceneKit/scn_metal>

// Default Sampler for the Skybox
constexpr sampler cubeSampler;


// Nodebuffer (you only need the enabled Matrix floats)
struct MyNodeBuffer {
    // float4x4 modelTransform;
    // float4x4 inverseModelTransform;
    float4x4 modelViewTransform; // required
    // float4x4 inverseModelViewTransform;
    float4x4 normalTransform; // required
    // float4x4 modelViewProjectionTransform;
    // float4x4 inverseModelViewProjectionTransform;
};

// Input Struct
typedef struct {
    float3 position [[ attribute(SCNVertexSemanticPosition) ]];
    float3 normal   [[ attribute(SCNVertexSemanticNormal)   ]];
} MyVertexInput;

// Struct filled by the Vertex Shader
struct SimpleVertexRefract
{
    float4 position [[position]];
    float  k;
    float3 worldSpaceReflection;
    float3 worldSpaceRefraction;
};

// VERTEX SHADER
vertex SimpleVertexRefract myVertexRefract(MyVertexInput in [[stage_in]],
                                          constant SCNSceneBuffer& scn_frame [[buffer(0)]],
                                          constant MyNodeBuffer& scn_node [[buffer(1)]])
{
float4 modelSpacePosition(in.position, 1.0f);
float4 modelSpaceNormal(in.normal, 0.0f);

// We'll be computing the reflection in eye space, so first we find the eye-space
// position. This is also used to compute the clip-space position below.
float4 eyeSpacePosition         = scn_node.modelViewTransform * modelSpacePosition;

// We compute the eye-space normal in the usual way.
float3 eyeSpaceNormal           = (scn_node.normalTransform * modelSpaceNormal).xyz;

// The view vector in eye space is just the vector from the eye-space position.
float3 eyeSpaceViewVector       = normalize(-eyeSpacePosition.xyz);

float3 view_vec                 = normalize(eyeSpaceViewVector);
float3 normal                   = normalize(eyeSpaceNormal);

const float ETA                 = 1.12f; // (this defines the intensity of the refraction. 1.0 will be no refraction)
float c                         = dot(view_vec, normal);
float d                         = ETA * c;
float k                         = clamp(d * d + (1.0f - ETA * ETA), 0.0f, 1.0f); // k is used in the fragment shader

// for Reflection / Refraction
// To find the reflection/refraction vector, we reflect/refract the (inbound) view vector about the normal.
float4 eyeSpaceReflection       = float4(reflect(-eyeSpaceViewVector, eyeSpaceNormal), 0.0f);
float4 eyeSpaceRefraction       = float4(refract(-eyeSpaceViewVector, eyeSpaceNormal, ETA), 0.0f);

// To sample the cube-map, we want a world-space reflection vector, so multiply
// by the inverse view transform to go back from eye space to world space.
float3 worldSpaceReflection     = (scn_frame.inverseViewTransform * eyeSpaceReflection).xyz;
float3 worldSpaceRefraction     = (scn_frame.inverseViewTransform * eyeSpaceRefraction).xyz;

// Fill the Out-Struct
SimpleVertexRefract out;
out.position                    = scn_frame.projectionTransform * eyeSpacePosition;
out.k                           = k;
out.worldSpaceReflection        = worldSpaceReflection; //
out.worldSpaceRefraction        = worldSpaceRefraction; //
return out;
}

// FRAGMENT SHADER
fragment float4 myFragmentRefract(SimpleVertexRefract in [[stage_in]],
                                  texturecube<float, access::sample> cubeTexture [[texture(0)]])
{
// Since the reflection vector's length will vary under interpolation, we normalize it
// and flip it from the assumed right-hand space of the world to the left-hand space
// of the interior of the cubemap.
float3 worldSpaceReflection     = normalize(in.worldSpaceReflection) * float3(1.0f, 1.0f, -1.0f);
float3 worldSpaceRefraction     = normalize(in.worldSpaceRefraction) * float3(1.0f, 1.0f, -1.0f);

float3 reflection               = cubeTexture.sample(cubeSampler, worldSpaceReflection).rgb;
float3 refraction               = cubeTexture.sample(cubeSampler, worldSpaceRefraction).rgb;

float4 color;
color.rgb                       = mix(reflection, refraction, float3(in.k)); // this is where k is finally used
color.a                         = 1.0f;
return color;
}

Compile and Run. The effect should look like this:

*If you use an AR Scene - setting the Skybox will overwrite the current camera feed, you might want to backup the AR feed elsewhere before you set the skybox, like so: Make a global definition:

var originalARSource : Any? = nil // screen Scene Backup
originalARSource = myScene.background.contents

You can jump back to the AR feed by setting myScene.background.contents back to originalARSource

** In ARKit make sure to set the Tracking Configuration to .none during the skybox is active:

configuration.environmentTexturing = .none

Answer 2

Yes, everything is possible with the amazing power of Physics! You'll need to create your own shader though. From Wikipedia:

In the field of computer graphics, a shader is a computer program that is used to do shading: the production of appropriate levels of color within an image, or, in the modern era, also to produce special effects or do video post-processing. A definition in layman's terms might be given as "a program that tells a computer how to draw something in a specific and unique way".

objc.io has a great tutorial on SceneKit if you're interested.