11

I have a RealityKit project in Xcode and I want to record the ARView. I considered ReplayKit, but that is for screen recording, I want to record only the ARView with its camera feed. I considered the open source project ARVideoKit by AFathi but that doesn't support RealityKit... something about different rendering paths. I have found a Medium article which describes how to implement a recording feature in an ARKit app, but the problem is that it requires the method: func renderer(_ renderer: SCNSceneRenderer) which is not available in RealityKit because it is specifically a SceneKit method.

dandan78
  • 13,328
  • 13
  • 64
  • 78
Tad
  • 889
  • 9
  • 23
  • You can use ReplayKit and then can crop the video before based on ARView's frame. Also you can eliminate the unwanted GUI elements on secondary window. – max9xs Jan 02 '20 at 11:34
  • I am in the same situation, but I will not use ReplayKit. Not only do you need to get around the fact it records UI elements (which is possible), but it asks your users for permission to record your screen (which is ugly). I am going to try and work out how to convert the YUV video output generated in the `session(_ session: ARSession, didUpdate frame: ARFrame)` delegate method into RGB and write to an AVCaptureSession in a similar way to how you would record video. – JCutting8 Jun 08 '20 at 07:12
  • @JCutting8 did you succeed? I'm also trying. – Jun Jul 15 '20 at 05:04
  • I am in the same situation. Still haven't find worked solutions? – Ruben Nahatakyan Sep 03 '20 at 07:09
  • Use ReplayKit, best solution! – Peter Pohlmann Jul 02 '21 at 08:40

1 Answers1

5

My answer assumes you are familiar with recording video and audio using AVAssetWriter.

There is a captured frame that is provided as part of the ARKit session(_:didUpdate:) method. The ARFrame object returned has a CVPixelBuffer named capturedFrame. Handle the frame as you would a regular video recording session, except instead of being captured in captureOutput(_:didOutput:from:) method, it is captured here instead. You may still need a captureOutput(_:didOutput:from:) method for audio if you intend on recording audio from the microphone, too.

In my case, I converted my captured frame into a MTLTexture and used Metal to process my video frames before passing them to an AVAssetWriter. I wanted to draw on top of my camera frames before recording. Unfortunately, doing this is very complicated and not a quick and short copy+paste answer I'm afraid. Hopefully pointing you to the capturedFrame object returned by ARKit is a good place for you to start.

Example on how to record videos using AVAssetWriter: https://programmersought.com/article/80131041234/;jsessionid=38CBA6743FB3C440DE9D2B25A6854B28

You will also need to verse yourself in Metal if you want to draw your 3D models into the capture feed before it is encoded to video: https://developer.apple.com/documentation/metalkit/

JCutting8
  • 732
  • 9
  • 29
  • I'm moderately experienced with recording videos due to having worked with a couple of related libraries on Github. Would you mind making some parts of your code open source? I'm currently working with https://github.com/Silence-GitHub/BBMetalImage in one project but my most recent client would prefer RealityKit for the camera feature. If not open source I'd be very happy to give you my email, too. Would really appreciate your help since I don't think I could stem it all on my own :) – Moritz Aug 10 '20 at 23:48
  • With this method, AR models are saving in the frame? Or just a video source? – Ruben Nahatakyan Sep 03 '20 at 07:05
  • @RubenNahatakyan Not sure because in my case i just needed the camera feed and then drew on top of it with Metal. So in my case, I just got the camera feed. – JCutting8 Sep 04 '20 at 12:46
  • 1
    @RubenNahatakyan - the `capturedFrame` that the `session(_:didUpdate:)` provides is the raw camera image and does not include the rendered AR content from your scene. – Michael Apr 07 '21 at 16:31
  • This will generate a video only of the video source any AR 3d Objects will NOT be displayed. – Peter Pohlmann Jul 02 '21 at 08:29
  • @PeterPohlmann Yes, this was already mentioned by Michael above. ARAnchors are provided in the delegate, you can draw these using metal if you so wish. – JCutting8 Jul 04 '21 at 02:10