4

How can I render a video as a diffuse texture on a node in SceneKit while using custom shader? This sounds like a simple thing to do but:

  • SceneKit elements don't support AVPlayerLayer as input texture (SO)
  • alternatives are:
    • using a SpriteKit scene with a SKVideoNode but:
      • Since iOS 10, rendering with openGL fails as soon as a SpriteKit scene is involved (SO) I've filled a radar
      • I could not figure out how to pass uniforms to a shader modifier when using Metal (SO) Things were working perfectly with openGL (on iOS 9, when SpriteKit was working) There's surprisingly very little documentation out there on that.
      • SCNProgram seems to fails as soon as a video is involved in the SceneKit scene (SO) Things work fine with the same setup without video
    • transforming the video output to a CGImage, passing it to a CALayer, and passing this layer to the SceneKit video node (no SpriteKit involved) This is detailed here, but not a viable solution as soon as the video is a bit large, because all the output is getting copied to image and this is costly..

I'm still quite reluctant to get out of SceneKit, since it's handling a lot of the annoying low level things with opengl/metal, but I can't get things to work right now...

As an additional bad news, it looks like SpriteKit has memory leak when used with Metal, and that it keeps alive the SceneView, and therefore all the parent views and everything attached to them...

You can find a reproduction of this issue here, and a simpler one here:

Community
  • 1
  • 1
Guig
  • 9,891
  • 7
  • 64
  • 126
  • 1
    Last year SpriteKit went one step forward (must have been something improved) and about three steps back (iOS 9). It took many months before Apple began addressing the problems. They didn't ever publicly acknowledge the problems despite many specific, recurring and matching complaints. – Confused Sep 17 '16 at 13:49
  • 1
    iOS 10 looks like it's SceneKit's turn to take a few steps back, and one forward (PBR). SceneKit has 1/20th (or less) of SpriteKit users. So it might take a whole year for them to get around to even looking at problems. Having said that, two of the guys that work on SceneKit are active in these forums. – Confused Sep 17 '16 at 13:50
  • 1
    Do you know how can I even have an idea if the issue is coming from SceneKit or SpriteKit? – Guig Sep 19 '16 at 16:37
  • 2
    Not really, depends too much on how you're doing things, and how obscure the right approach is to achieving this. The Apple documentation for Sprite Kit is poor. The documentation for SceneKit is atrocious. – Confused Sep 19 '16 at 16:40
  • That's what I fear. We're in the total darkness... – Guig Sep 19 '16 at 16:47
  • 2
    It's no direct help but... if you're into it, one more commit and we could start documenting anything we learn about SceneKit. I'm not a big believer in SO.Docs, but this is one use case it might be good for: http://stackoverflow.com/documentation/scenekit/commit – Confused Sep 19 '16 at 18:01
  • `As an additional bad news, it looks like SpriteKit has memory leak when used with Metal, and that it keeps alive the SceneView, and therefore all the parent views and everything attached to them...` can you prove this? – Knight0fDragon Sep 21 '16 at 15:06
  • yeah, let me put up a reproduction – Guig Sep 21 '16 at 18:14
  • Here is a very simple gist: https://gist.github.com/gsabran/30714b7c82ad24d43fe7179d638bbe47, and here is a more complex repo with description: https://github.com/gsabran/SceneKit-SpriteKit-memoryLeak – Guig Sep 22 '16 at 08:21
  • "The documentation for SceneKit is atrocious." You must be crazy. SceneKit's documentation is some of the best and most thorough I've ever seen. – CIFilter Oct 13 '16 at 00:12
  • @LucasTizma this made my day. Their documentation is almost not more informative than what the autocompletion tells you and I've wasted hours on it. – Guig Oct 13 '16 at 00:44
  • I wonder if you'd be able to extract the frames directly from the video source using AV Foundation and use a Metal texture buffer pool using `CVMetalTextureCacheCreate(_:_:_:_:_:)`. SceneKit materials can accept `MTLTexture`s directly. Bit of overhead, but still... – CIFilter Nov 30 '16 at 00:46
  • You can get a reference to the pixelBuffer using `copyPixelBufferForItemTime`. With that you should be able to create a texture. What is a bit frustrating though, is that once you use FairPlay protection Apple seems to have designed their API in a way so that there is no way you can display the decrypted video media outside of Apple's Ecosystem: http://stackoverflow.com/q/42839831/1097106 – schmittsfn Mar 18 '17 at 11:14

1 Answers1

2

Not being able to solve this issue, I wrote my own library that mimics some functionalities of SceneKit: https://github.com/team-pie/DDDKit

Guig
  • 9,891
  • 7
  • 64
  • 126
  • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - [From Review](/review/low-quality-posts/15212577) – Jakub Truhlář Feb 15 '17 at 00:50
  • 1
    I'm the author. Putting it here provides more visibility to an alternative that, while not being a direct answer to the question, solves the issue for people who are only using SceneKit to solve the original problem of 3D rendering with shaders. I didn't accept this answer and will not do so. When someone provides a complete answer, I'll remove this one which is the best answer to date. If you've a better answer please go ahead! – Guig Feb 15 '17 at 01:33
  • @Guig Looks like a great library - ignore the Stack Overflow police, they are hopeless. – Jordan Jan 12 '18 at 08:46
  • It might need a bit of a refreshment. Let me know if you run into issues! – Guig Jan 19 '18 at 00:02