1

I have a data array of Int16 or Int32 numerical values that are the raw image data from a 11MP camera chip with an RGGB pixel layout (CFA). The data are exported by the camera driver as FITS data, which is basically a vector or long string of bytes or 16bit/pixel data in my case.

I like to convert these data into a raw image format in Swift in order to use the powerful debayering and demosaicing features and algorithms in iOS/Swift. I do not intend to demosaic myself, since iOS has a great library for this already (see WWDC2016 keynote on Raw Processing with Core Image).

I need to make iOS “believe” my data are actual raw image data.

I tried using CreatePixelBufferWithBytes in Swift and then CIImage from pixelbuffer but to no avail. The CIImage.cgimage is not an RGB color image.

Is there a simple way to create a raw or DNG image in Swift from raw numerical data?

Here is what I tried with the CVPixelBuffer approach, but I do not get any color image out of this:

imgRawData is a [Int32] or [Float32] array with width*height number of elements.

var pixelBuffer: CVPixelBuffer?
let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
            kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue ]
    
CVPixelBufferCreateWithBytes(kCFAllocatorDefault, width, height, kCVPixelFormatType_14Bayer_RGGB, &imgRawData, 2*width, nil, nil, attrs as CFDictionary, &pixelBuffer)

let dummyImg = UIImage(systemName: "star.fill")?.cgImage
    
let ciiraw = CIImage(cvPixelBuffer: pixelBuffer!)
    
let cif = CIFilter.lanczosScaleTransform()
cif.scale = 0.25
cif.inputImage  = ciiraw
let cii = cif.outputImage
    
let context: CIContext = CIContext.init(options: nil)
guard let cgi = context.createCGImage(cii!, from: cii!.extent) else { return dummyImg! }

Quickview of Xcode shows me only black&white or grayscale images. So does the SwiftUI View of the CGImage...

Planetoid30
  • 21
  • 3
  • did you see this one question? Probably it will help https://stackoverflow.com/q/51372245/10595176 – Mol0ko Feb 08 '21 at 12:09
  • Thank you. I will check it out! – Planetoid30 Feb 08 '21 at 23:17
  • I checked it out... This is close, but not really the same. In that case the image is already available as RGBA, i.e. de-mosaiced. I want iOS to do the de-mosaicing of my raw image data (RGGB-CFA). – Planetoid30 Feb 08 '21 at 23:27

1 Answers1

0

You can use CGContext and pass your raw values in as bitmapinfo, see init:

init?(data: UnsafeMutableRawPointer?, width: Int, height: Int, bitsPerComponent: Int, bytesPerRow: Int, space: CGColorSpace, bitmapInfo: UInt32)

And for space parameter, which takes CGColorSpace you would use CGColorSpaceCreateDeviceRGB().

You will then use your image with a code similar to this one:

let imageRef = CGContext.makeImage(context!)
let imageRep = NSBitmapImageRep(cgImage: imageRef()!)

Play around with it for a bit, I think you will find what you are looking for.

Ctibor Šebák
  • 133
  • 1
  • 14
  • Thank you! How does CGContext know about the CFA, i.e. the pixel layout? It has to know that it is RGGB. This will also generate a CGImage and not a CoreImage, which is necessary for the raw image CIFilters, I believe. – Planetoid30 Feb 08 '21 at 09:29
  • http://wiki.hawkguide.com/wiki/Swift:_Convert_between_CGImage,_CIImage_and_UIImage use this guide when converting between formats – Ctibor Šebák Feb 08 '21 at 09:49
  • Thank you... I tried some of this. The issue is still that neither CG nor CI images know the pixel format, the RGGB. Converting between the different image representations are straight forward. – Planetoid30 Feb 08 '21 at 10:13
  • Can BitmapInfo be used for the RGGB pixelformat (CFA)? – Planetoid30 Feb 08 '21 at 10:17
  • Please see this: https://developer.apple.com/documentation/imageio/kcgimagepropertyexifcfapattern – Ctibor Šebák Feb 08 '21 at 10:57
  • Great. I will try that. – Planetoid30 Feb 08 '21 at 12:32
  • Will iOS/Swift use the EXIF data for the raw image processing? Or is it just info? – Planetoid30 Feb 08 '21 at 16:36
  • How do your apply the CGImageProperties to the CGImage? – Planetoid30 Feb 09 '21 at 21:04
  • 1
    Figured it more or less out. As described above, "RGGB" is the CFA of the sensor and not the CG colourspace. Therefore, the CFA layout has to be provided as "tagged info" in the raw data for CIRAWFilter. Raw image data are tagged! This is what I had been missing before. TIFF is a tagged format and is used for raw images. I would need to convert the image sensor properties to "tags" and then add the tagged data to the raw image data in a single data variable, before I read and convert them with CIRAWFILTER to a CI image. – Planetoid30 Jul 09 '22 at 16:58