0

I am attempting to build an application on IOS using gluon mobile that uses the devices camera to take a picture then send the file data of that picture to a backend server. However, this cannot be done directly since the PictureService provided by gluon only returns a javafx Image object when a picture is taken.

To get around this I am attempting to get the raw pixel data from the Image object IOSImageService.java, then use UIKit and Objective-C to take advantage of IOS native libraries to convert the raw pixel data into a JPEG file format JPEGConversion.m(code mainly taken from here).

From my DEBUG OUTPUT I can see that when I attempt to get the NSData object from the UIImage object it always comes out NULL and with a length of 0. I am not sure where I am going wrong with this code.

Am I using the UIKit libraries wrong? Am I not using Objective-C correctly? Or, am I not even sending to data to the Objective-C code in the proper format?

Everything here seems like I am going in the proper direction, I just have to successfully get the NSData for the JPEG file and then convert that into a jbyteArray and return it.

Java: IOSImageService.java

package com.gluonhq.charm.down.plugins.ios;

import com.gluonhq.charm.down.plugins.ImageService;
import javafx.scene.image.Image;
import javafx.scene.image.PixelFormat;
import javafx.scene.image.PixelReader;

public class IOSImageService implements ImageService{

static {
    System.loadLibrary("JPEGConversion");
}

private native byte[] nativeGetImageBytes(byte[] array, int width, int height);

@Override
public byte[] getImageBytes(Image image) {
    int width = (int)image.getWidth();
    int height = (int)image.getHeight();

    System.out.println("Width: " + Integer.toString(width) + "\nHeight: " + Integer.toString(height));

    //Get raw pixel data from image
    byte[] BGRAFormattedArray = new byte[width * height * 4];
    PixelReader p = image.getPixelReader();
    p.getPixels(0, 0, width, height, PixelFormat.getByteBgraPreInstance(), BGRAFormattedArray, 0, width * 4);
    System.out.println("Size: " + Integer.toString(BGRAFormattedArray.length));

    //Change format from BGRA to RGBA
    byte[] RGBAFormattedArray = new byte[width * height * 4];
    for(int i=0;i<BGRAFormattedArray.length;i+=4) {
        RGBAFormattedArray[i + 0] = BGRAFormattedArray[i + 2];
        RGBAFormattedArray[i + 1] = BGRAFormattedArray[i + 1];
        RGBAFormattedArray[i + 2] = BGRAFormattedArray[i + 0];
        RGBAFormattedArray[i + 3] = BGRAFormattedArray[i + 3];
    }

    return nativeGetImageBytes(RGBAFormattedArray, width, height);
}

}

Objective-C: JPEGConversion.m

JNIEXPORT jbyteArray JNICALL Java_com_gluonhq_charm_down_plugins_ios_IOSImageService_nativeGetImageBytes
(JNIEnv *env, jobject obj, jbyteArray array, jint width, jint height)
{
jbyte* elements = (*env)->GetByteArrayElements(env, array, NULL);
if(elements == NULL) {
    NSLog(@"RETURNING BECAUSE DATA WAS NULL");
    return NULL;
}

int length = (*env)->GetArrayLength(env, array);

int w = (int)width;
int h = (int)height;

NSLog(@"ARRAY LENGTH: %i", length);
NSLog(@"Width: %i", w);
NSLog(@"Height: %i", h);

CGDataProviderRef provider = CGDataProviderCreateWithData(NULL, elements, (64 * 64 * 4), NULL);

int bitsPerComponent = 8;
int bitsPerPixel = 32;
int bytesPerRow = 4 * w;

CGColorSpaceRef colorSpaceRef = CGColorSpaceCreateDeviceRGB();
CGColorRenderingIntent renderingIntent = kCGRenderingIntentDefault;
CGImageRef imageRef = CGImageCreate(w, h, bitsPerComponent, bitsPerPixel, bytesPerRow, colorSpaceRef, kCGBitmapByteOrderDefault | kCGImageAlphaLast, provider, NULL, false, renderingIntent);


size_t b_width = CGImageGetWidth(imageRef);
NSLog(@"IMAGE REFFERENCE WIDTH: %i", (int) b_width);

if(imageRef == NULL) {
    NSLog(@"RETURNING BECAUSE IMAGE REFERENCE IS NULL");
    return NULL;
}

UIImage *image = [UIImage imageWithCGImage:imageRef];

if(image == NULL) {
    NSLog(@"UIIMAGE WAS NULL");
}else {
    NSLog(@"UIIMAGE WAS NOT NULL");
}

NSData* pictureData = UIImageJPEGRepresentation(image, 1.0);

if(pictureData == NULL) {
    NSLog(@"PICTURE DATA NULL");
}

NSLog(@"bytes in hex: %@", [pictureData description]);

//int picLength = (int)[pictureData length];
NSLog(@"Picture Length: %ld", [pictureData length]);

CGImageRelease(imageRef);
(*env)->ReleaseByteArrayElements(env, array, elements, JNI_ABORT);

//dummy return value
jbyte a[] = {1,255,1,31,1,15};
jbyteArray result = (*env)->NewByteArray(env, 6);
(*env)->SetByteArrayRegion(env, result, 0, 6, a);
return result;

}

DEBUG OUTPUT

Width: 128
Height: 128
Size: 65536
2018-03-14 15:51:22.117174-0230 MultiplatformGluonApplicationApp[1166:1138266] ARRAY LENGTH: 65536
2018-03-14 15:51:22.117270-0230 MultiplatformGluonApplicationApp[1166:1138266] Width: 128
2018-03-14 15:51:22.117349-0230 MultiplatformGluonApplicationApp[1166:1138266] Heigth: 128
2018-03-14 15:51:22.117413-0230 MultiplatformGluonApplicationApp[1166:1138266] IMAGE REFFERENCE WIDTH: 128
2018-03-14 15:51:22.117518-0230 MultiplatformGluonApplicationApp[1166:1138266] UIIMAGE WAS NOT NULL
2018-03-14 15:51:22.119407-0230 MultiplatformGluonApplicationApp[1166:1138266] PICTURE DATA NULL
2018-03-14 15:51:22.119504-0230 MultiplatformGluonApplicationApp[1166:1138266] bytes in hex: (null)
2018-03-14 15:51:22.119537-0230 MultiplatformGluonApplicationApp[1166:1138266] Picture Length: 0
JPG size: 6
[ 0x01 0xFF 0x01 0x1F 0x01 0x0F ]
José Pereda
  • 44,311
  • 7
  • 104
  • 132
colio303
  • 81
  • 10
  • 1
    It seems to me that your proposal takes the following path: (pictures service) native iOS File and Image, converted and _scaled down_ to a JavaFX image, then send that image back to iOS (image service), to get a file, and back to Java. Since you are getting all the troubles to create a new service, with native code on iOS, wouldn't be more straightforward to modify the current PicturesService adding some API to return directly the `File`? – José Pereda Mar 14 '18 at 19:43
  • 1
    In case you haven't checked it yet, see this [answer](https://stackoverflow.com/a/42052139/3956070), and this other [answer](https://stackoverflow.com/a/47581161/3956070), that directly try to work with the JavaFX Image. – José Pereda Mar 14 '18 at 19:57
  • @José Pereda Thank you for your suggestion to modify the original PictureService, that seemed to be the way to go. I just had to get the service to stop one step short of creating the Image object and just return the raw file bytes. Then I compiled my own jniLib where the UIImage gets converted to a JPEG instead of a PNG file. – colio303 Mar 15 '18 at 00:38

0 Answers0