Load AVAsset video in the background and replace playing placeholder video once it's playable in Swift and RealityKit

Question

I'm using the following code to create a video player for detected reference images in AR session. Currently I display a placeholder video and after 1 second switch to real video that I want played. However, I would like to show the placeholder video until the real video is ready to be played.

I tried experimenting with AVAsset and observing the playable status based on this: Knowing when AVPlayer object is ready to play - however I didn't have any success.

        func createVideoNode(_ target: ARReferenceImage) -> ModelEntity {
        var videoPlane = ModelEntity()
        var targetName: String = ""
        
        if let name = target.name,
           let validURL = URL(string: "https://testdomain.com/\(name).mp4") {
            targetName = name

            // Use the preloaded placeholder asset to create an AVPlayer
            if let placeholderAsset = parent.placeholderAsset {
                let placeholderPlayer = AVPlayer(playerItem: AVPlayerItem(asset: placeholderAsset))
                let videoMaterial = VideoMaterial(avPlayer: placeholderPlayer)
                videoPlane = ModelEntity(mesh: .generatePlane(width: Float(target.physicalSize.width), depth: Float(target.physicalSize.height)), materials: [videoMaterial])
                placeholderPlayer.play()

                DispatchQueue.global(qos: .background).async {
                    let videoPlayer = AVPlayer(url: validURL)
                    NotificationCenter.default.addObserver(forName: .AVPlayerItemDidPlayToEndTime, object: videoPlayer.currentItem, queue: .main) { [weak videoPlayer] _ in
                        videoPlayer?.seek(to: CMTime.zero)
                        videoPlayer?.play()
                    }
                    DispatchQueue.main.asyncAfter(deadline: .now() + 1.0) {
                        let videoMaterial = VideoMaterial(avPlayer: videoPlayer)
                        videoPlane.model?.materials = [videoMaterial]
                        videoPlayer.play()
                        self.parent.videoPlayers[targetName] = videoPlayer
                        print (target.name as Any)
                    }
                }
            } else {
                fatalError("Failed to load placeholder video asset.")
            }
        }

        return videoPlane
    }

score 2 · Accepted Answer · answered Jun 01 '23 at 19:39

The key to resolving this issue is making sure the AVPlayer's item is actually ready to play before switching the video. You can use the Key-Value Observing (KVO) on the AVPlayerItem's status property to get notified when it's ready to play.

Here is the updated createVideoNode(_:) function:

func createVideoNode(_ target: ARReferenceImage) -> ModelEntity {
    var videoPlane = ModelEntity()
    var targetName: String = ""
    
    if let name = target.name,
       let validURL = URL(string: "https://testdomain.com/\(name).mp4") {
        targetName = name

        // Use the preloaded placeholder asset to create an AVPlayer
        if let placeholderAsset = parent.placeholderAsset {
            let placeholderPlayer = AVPlayer(playerItem: AVPlayerItem(asset: placeholderAsset))
            let videoMaterial = VideoMaterial(avPlayer: placeholderPlayer)
            videoPlane = ModelEntity(mesh: .generatePlane(width: Float(target.physicalSize.width), depth: Float(target.physicalSize.height)), materials: [videoMaterial])
            placeholderPlayer.play()

            DispatchQueue.global(qos: .background).async {
                let asset = AVAsset(url: validURL)
                let playerItem = AVPlayerItem(asset: asset)
                let videoPlayer = AVPlayer(playerItem: playerItem)

                // Observe the status of playerItem.
                playerItem.addObserver(self, forKeyPath: "status", options: .new, context: nil)

                NotificationCenter.default.addObserver(forName: .AVPlayerItemDidPlayToEndTime, object: videoPlayer.currentItem, queue: .main) { [weak videoPlayer] _ in
                    videoPlayer?.seek(to: CMTime.zero)
                    videoPlayer?.play()
                }

                self.parent.videoPlayers[targetName] = videoPlayer
            }
        } else {
            fatalError("Failed to load placeholder video asset.")
        }
    }

    return videoPlane
}

// Add this method to handle observed value change
override func observeValue(forKeyPath keyPath: String?, of object: Any?, change: [NSKeyValueChangeKey : Any]?, context: UnsafeMutableRawPointer?) {
    if keyPath == "status" {
        if let playerItem = object as? AVPlayerItem, playerItem.status == .readyToPlay {
            DispatchQueue.main.async { [weak self] in
                if let videoPlane = self?.videoPlane {
                    let videoMaterial = VideoMaterial(avPlayer: playerItem.player)
                    videoPlane.model?.materials = [videoMaterial]
                    playerItem.player?.play()
                }
            }
        }
    }
}

This version of the function now creates an AVPlayerItem using the AVAsset. It then adds the ViewController as an observer of the playerItem's status property. The observeValue(forKeyPath:of:change:context:) method gets called when the status changes. When the status is .readyToPlay, it switches the video on the main queue.

Please note that the observeValue method is a standard method for classes that inherit from NSObject, make sure your class does that. Also remember to remove the observer when it's no longer needed.

You will also have to hold a strong reference to your AVPlayerItem and AVPlayer in order to observe changes. This might necessitate some architectural changes (adding properties to your class).

This solution should give you a general direction, but you might need to adjust it to fit your specific project setup and requirements.

Load AVAsset video in the background and replace playing placeholder video once it's playable in Swift and RealityKit

1 Answers1