0

How to draw into CIImage (or maybe into CVPixelBuffer, but I guess it easier to add text to CIImage)? not to UIImage

I record video (.mp4 file) using AVAssetWriter and CMSampleBuffer (from video, audio inputs). While recording I want to add text on the video frames, I'm already converting CMSampleBuffer to CIImage, now I need somehow add text to CIImage and convert back to CVPixelBuffer. I didn't really find any simple examples in Swift how to add (draw) text to CIImage or add anything else (like lines, rects, not just text).

/// encode video buffer
func appendVideoSampleBuffer(_ sampleBuffer: CMSampleBuffer) {

    // before encoding video frames into a file I would like process it
    // 1. convert CMSampleBuffer to ciImage

    let cvPixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
    let ciimage : CIImage = CIImage(cvPixelBuffer: cvPixelBuffer)

    // 2. do something with CIImage


    // 3. convert CIImage back to CMSampleBuffer

    let sampleBufferWithText == ...


    appendSampleBuffer(sampleBufferWithText, ofMediaType: AVMediaType.video.rawValue)
}

/// encode audio buffer
func appendAudioSampleBuffer(_ sampleBuffer: CMSampleBuffer) {
    appendSampleBuffer(sampleBuffer, ofMediaType: AVMediaType.audio.rawValue)
}
user924
  • 8,146
  • 7
  • 57
  • 139
  • @dfd then maybe somehow add text directly to CVPixelBuffer/CMSampleBuffer? – user924 Mar 01 '18 at 13:31
  • Good, then the confusion was on my part. :-) I haven't worked much with buffers, but for a CIImage the concept is to create a UILabel, then using a UI graphics context turn it into an image, getting a CIImage from it, and then a CIFilter to merge with it. In *Core Image for Swift* chapter 2 Simon Gladman describes using `CIHieghtFieldFromMask` and `CIShadedMaterial` to do something close to what I'm talking about.... –  Mar 01 '18 at 13:44
  • Here's the specific playground for it: https://github.com/FlexMonkey/CoreImageForSwiftPlaygrounds/tree/master/Chapter2.playground/Pages/Height%20Field%20%26%20Shaded%20Material.xcplaygroundpage Also, you might want to check out other projects he has, including one for live camera filtering. Most are written in Swift 2 though, so you'll either need Xcode 8 or know how to manipulate Xcode 9 project settings to upgrade things to run them. His techniques are the important part though, and they still work. –  Mar 01 '18 at 13:47
  • @dfd but I think performance won't be that good (with UILabel), I also experimented with UIImage (which I don't use) https://stackoverflow.com/questions/28906914/how-do-i-add-text-to-an-image-in-ios-swift and adding text to it takes a lot of cpu time – user924 Mar 01 '18 at 13:48
  • @dfd if creating CIImage from a text and blending it with frame CIImage using `CIBlendWithMask` will be fast then would be great, but problem how to convert CIImage back to CMSampleBuffer? – user924 Mar 01 '18 at 14:05
  • @dfd yes combining two CIImage works pretty fast, but have to find out now how to send buffer from new combined CIImage to CMSampleBuffer – user924 Mar 01 '18 at 15:37

1 Answers1

0

You can create NSAttributedString for text -> draw this string in to cgContext -> convert UIImage/CGImage -> CIImage

Trần Minh Quang
  • 81
  • 1
  • 3