1

I am looking for a way to convert a byte array to CIIimage so that I can feed it into a ML model for classification. I am using a REST HTTP server, where I send a POST request to the server with payload as the image. The image bytes received by the server needs to be processed and converted into a CIImage format for MAC OS so that it can be fed into a ML model which accepts requests of type VNImageRequestHandler(ciImage: <ciimage>).

Could someone give an example to do this in swift ?

VNImageRequestHandler : NSObject

let data = Data(bytes) let imgHandler = VNImageRequestHandler(ciImage: data) The above data variable needs to be typecased to be of type CIImage.

On the HTTP server side, I am receiving the bytes of the image like this: imageData = request.body.bytes

mredig
  • 1,736
  • 1
  • 16
  • 29
psbits
  • 1,787
  • 5
  • 19
  • 34
  • Are you using the MNIST database by any chance? – Code Different Jul 17 '18 at 02:59
  • Nopes. Any interesting things with MNIST ? – psbits Jul 17 '18 at 03:40
  • What is the type of bytes returned from REST. Are they raw bytes, 64based encoded or something? Can you tell for more detail. – Cosmos Man Jul 17 '18 at 05:41
  • If they are raw bytes, convert to CGImage->CIImage. If they are 64base encoded, convert to Data->NSImage->CIImage. – Cosmos Man Jul 17 '18 at 05:44
  • @CosmosMan, I am just sending the bytes of the image using curl to the REST server and then I am trying to covert them into CIImage. Since I don't pass any headers, the default content-type returned is `application/x-www-form-urlencoded`. Even if I set the headers as 'Content-type: image/jpeg" while making the curl request, the bytes received by the server are the same but converting it to NSImage gives me a 0*0 image. – psbits Jul 17 '18 at 09:25
  • @CosmosMan Also, I am receiving the bytes on the HTTP server like this: imageData = request.body.bytes and these are just raw bytes. – psbits Jul 17 '18 at 09:45
  • @psbits If they are raw bytes, converting them to NSData or Data will not contain any information(like these bytes represents JPEG, PNG, BMP, etc). Raw bytes(4bytes per pixel) should be handled with CGImage since CGImage has many way to represent raw bytes (rgba, bgra , etc) with things like Bytes per row, bytes per pixel, bits per pixel, color space, size, etc. If you have obtained CGImage, you can convert to CIImage with CIImage.init(CGImage:). Can you print and comment the first 20 characters of your raw bytes? – Cosmos Man Jul 17 '18 at 09:59
  • @CosmosMan interesting. Sure, it's just a simple cat image which has the following raw bytes: `[255, 216, 255, 225, 108, 110, 115, 58, 114, 100, 102, 61, 34, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 119, 51, 46, 111, 114, 103, 47, 49, 57, 57, 57, 47, 48, 50, 47, 50, 50, 45, 114, 100, 102, 45, 115, 121, 110, 116, 97, 120, 45, 110, 115, 35, 34, 62, 32, 60, 114, 100, 102, 58, 68, 101, 115, 99, 114, 105, 112, 116, 105, 111, 110, 32, 114, 100, 102, 58, 97, 98, 111, 117, 116, 61, 34, 34, 32, 120, 109, 108, 110, 115, 58, 120, 109, 112, 82, 105, 103, 104, 116, 115, 61, 34, 34, 32, ........]` – psbits Jul 17 '18 at 10:08
  • @CosmosMan I am completely new to this. Could you share a sample code snippet to do this ? Converting from raw bytes -> CGImage -> CIImage. – psbits Jul 17 '18 at 10:09
  • @psbits Initializing from CGImage is a bit more complex than NSImage. Have a look at this [post](https://stackoverflow.com/questions/21477402/coverting-a-byte-array-to-cgimage). – Cosmos Man Jul 17 '18 at 10:56
  • @psbits You can check my answer too. Since the link is written in ObjC, I made an answer with Swift. – Cosmos Man Jul 17 '18 at 11:28
  • @CosmosMan Thanks. I actually followed this: https://stackoverflow.com/questions/2261177/cgimage-from-byte-array and was able to get something working. Would you know how can I extract the dimensions of the image from the byte array ? In this post, we always create a 64*64*3 matrix. Ideally I would like to regenerate the exact same image as I have originally. – psbits Jul 17 '18 at 11:34
  • As far as I know, it's impossible to extract width and height of image from raw buffer. It has to be explicitly stated at somewhere. For example, in JPG or PNG they have some field that retain the size information of image. I'm sure you must get the size information from somewhere rather than raw byte array. – Cosmos Man Jul 17 '18 at 11:39
  • I'll try to figure this out and update this post with my findings on this. Thank you so much for your inputs @CosmosMan. – psbits Jul 17 '18 at 11:44
  • @CosmosMan it is possible to do it in java: https://stackoverflow.com/questions/17189129/extract-images-width-height-color-and-type-from-byte-array. I think there should be a way to do it in swift too. – psbits Jul 17 '18 at 11:56

1 Answers1

1

Convert byte array to CGImage using this method. You must make it sure that your bytes are rgba 32 bit pixel raw bytes.

func byteArrayToCGImage(raw: UnsafeMutablePointer<UInt8>, // Your byte array
                        w: Int, // your image's width
                        h: Int // your image's height
    ) -> CGImage! {

    // 4 bytes(rgba channels) for each pixel
    let bytesPerPixel: Int = 4
    // (8 bits per each channel)
    let bitsPerComponent: Int = 8

    let bitsPerPixel = bytesPerPixel * bitsPerComponent;
    // channels in each row (width)
    let bytesPerRow: Int = w * bytesPerPixel;

    let cfData = CFDataCreate(nil, raw, w * h * bytesPerPixel)
    let cgDataProvider = CGDataProvider.init(data: cfData!)!

    let deviceColorSpace = CGColorSpaceCreateDeviceRGB()

    let image: CGImage! = CGImage.init(width: w,
                                       height: h,
                                       bitsPerComponent: bitsPerComponent,
                                       bitsPerPixel: bitsPerPixel,
                                       bytesPerRow: bytesPerRow,
                                       space: deviceColorSpace,
                                       bitmapInfo: [],
                                       provider: cgDataProvider,
                                       decode: nil,
                                       shouldInterpolate: true,
                                       intent: CGColorRenderingIntent.defaultIntent)



    return image;
}

Using this method, you can convert to CIImage like this.

let cgimage = byteArrayToCGImage(raw: <#Pointer to Your byte array#> ,
                                 w: <#your image's width#>,
                                 h: <#your image's height#>)
if cgimage != nil {
    let ciImage = CIImage.init(cgImage: cgimage)
}

According to the comment, your datas might be RGB raw bytes rather than RGBA. In this case, you will have to allocate new buffer, put 255 for each alpha channel manually and pass that buffer to the method.

Updated for convertion of 32 bits RGB to 32 bits RGBA

func convertTo32bitsRGBA(from32bitsRGB pointer: UnsafeMutablePointer<UInt8>!,
                         width: Int,
                         height: Int) -> UnsafeMutablePointer<UInt8> {

    let pixelCount = width * height
    let memorySize = pixelCount * 4
    let newBuffer = malloc(memorySize).bindMemory(to: UInt8.self, capacity: width * height)

    var i = 0;
    while(i < pixelCount) {
        let oldBufferIndex = i * 3;
        let newBufferIndex = i * 4;

        // red channel
        newBuffer.advanced(by: newBufferIndex).pointee = pointer.advanced(by: oldBufferIndex).pointee
        // green channel
        newBuffer.advanced(by: newBufferIndex + 1).pointee = pointer.advanced(by: oldBufferIndex + 1).pointee
        // blue channel
        newBuffer.advanced(by: newBufferIndex + 2).pointee = pointer.advanced(by: oldBufferIndex + 2).pointee
        // alpha channel
        newBuffer.advanced(by: newBufferIndex + 3).pointee = 0xff;

        // &+ is used for little performance gain
        i = i &+ 1;
    }


    return newBuffer;
}

You can call the converter method with your rgb image buffer as follow

let newImageBuffer = convertTo32bitsRGBA(from32bitsRGB: <#Your RGB image buffer#>,
                    width: <#Your image pixel row count or width#>,
                    height: <#Your image pixel column count or height#>)

but remember, like in C, C++ or Objective-C, you are responsible to release the memory allocation returned by this method. These are pointers which memory are not managed by compiler.

You can release with simple function.

newImageBuffer.deallocate();

After deallocation, you must not access the deallocated memory. If you do so, you will get BAD_ACCESS_EXC (Bad access exception thrown by OS for accessing memory you do not own).

Cosmos Man
  • 593
  • 3
  • 15
  • Thank you so much @CosmosMan. This is more or less I did as well. Could you please share a code snippet to convert RBB byte array to RGBA in swift ? – psbits Jul 17 '18 at 11:43
  • Also, if I set bytesPerPixel to 3 and use RGB instead , would I need to convert it to RGBA ? – psbits Jul 17 '18 at 11:47
  • @psbits I've updated the answer with what you have requested. The code I gave here will suffer from performance drop if you process large image. If you wish, I can give you code boosted with GCD(Grand Central Dispatch) concurrent programming. – Cosmos Man Jul 17 '18 at 14:03
  • Thank you @CosmosMan. Sure. That will be great but before that my question is that what's the drawback if I use RGB ( 3 bytes per pixel) instead of RGBA (4 bytes per pixel) ? – psbits Jul 17 '18 at 15:06
  • @psbits It is just a representation. Since JPG has no alpha channel, they can be used to represent camera outputs(which does not require alpha channel). PNG does have alpha channel so that it can be used as icons or images in app that can blend with other UI elements. But JPG or PNG might not represent these information as raw pixels but compressed format (I don’t know exactly how they are formatted). On the other hand, bitmap use raw bytes and along with some other attributes(width , height, bytes per row, etc) but they are huge in size compared to JPG and PNG but faster at processing – Cosmos Man Jul 18 '18 at 02:36
  • @psbits UIKit represent 4 channel raw pixels as backing store I think since they wont want to make compression or depression at the time of need such as animations, transition, etc. – Cosmos Man Jul 18 '18 at 02:42