How to convert Flutter CameraImage to a Base64-encoded binary data object from an Image Stream

Question

With the introduction of an image stream in the flutter camera version 0.2.8, I've tried to integrate it into my project to use alongside AWS.

Amazon requires the image is in the format:

Blob of image bytes up to 5 MBs.
Type: Base64-encoded binary data object
Length Constraints: Minimum length of 1. Maximum length of 5242880.

Previously I used the Camera package to take a picture, load the picture and then convert it as amazon required, but using an ImageStream is much more suited for what I'd like to do. My previous approach was:

// Take the picutre
await _cameraController.takePicture(path);

// Load it from my filesystem
File imagefile = new File(path); 

// Convert to amazon requirements
List<int> imageBytes = imagefile.readAsBytesSync();
String base64Image = base64Encode(imageBytes);

However, using an image stream, I cannot find any easy way to convert a CameraImage to the format that amazon requires. I don't have much experience with images so I'm quite stuck.

I attempted to manipulate the code used in the firebase ml & camera stream demo

final int numBytes =
    image.planes.fold(0, (count, plane) => count += plane.bytes.length);
final Uint8List allBytes = Uint8List(numBytes);

int nextIndex = 0;
for (int i = 0; i < image.planes.length; i++) {
  allBytes.setRange(nextIndex, nextIndex + image.planes[i].bytes.length,
      image.planes[i].bytes);
  nextIndex += image.planes[i].bytes.length;
}

// Convert as done previously
String base64Image = base64Encode(allBytes);

However, AWS responded with a InvalidImageFormatException. If someone knows how to correctly encode the image that would be awesome! Thanks

Why do you think that `CameraImage` is better for you? It gives access to uncompressed pixel data, whereas it seems that AWS needs compressed, encoded data like PNG or JPG. ("Amazon Rekognition supports the PNG and JPEG image formats. That is, the images you provide as input to various API operations, such as DetectLabels and IndexFaces must be in one of the supported formats.") What benefit are you trying to achieve by switching from the way that works to this new way? If you start with raw bytes you will need to do the compression/encoding yourself - which is notoriously slow. — Richard Heap, Jan 17 '19 at 15:55
@RichardHeap The reason I want to use the `CameraImage` is because I'm using both AWS and firebase ml to do facial detection and then matching. Firebase ml allows me to detect whether there is a face as done in the [example I linked in my post](https://github.com/bparrishMines/mlkit_demo/). After detecting a face from the `CameraImage` I then want to use that same photo, convert it to Amazon's requirements for [matching a face](https://docs.aws.amazon.com/rekognition/latest/dg/API_SearchFacesByImage.html). Taking a picture afterwards may result in the person moving and being out of frame :( — Alexander Blyth, Jan 17 '19 at 23:16
The `CameraImage` that you are getting is almost certainly in YUV 420 format (check `cameraImage.format.group` to confirm). Converting that to RGB and then PNG or JPEG is non-trivial, and is probably done best in native code. For example Android has a `YUVImage` class that has a `compressToJpeg` method. Trying to do that in pure Dart will likely be very s-l-o-w. You will likely end up writing a plugin to call the native Android and iOS functions. There's am interesting article [here](https://blog.jayway.com/2018/09/27/face-recognition-with-aws-and-android-things/). — Richard Heap, Jan 18 '19 at 00:44
Please check your `base64Image` string on the console, make sure it starts with data image data type, for example `data:image/png;base64,****base64`. If it doesn't start with it you should add it manually — Putra Ardiansyah, Jan 28 '19 at 07:17
it does seem camera image is YUV 420. Sorry for bumping an old post, but if anyone has come up with a clever solution to this problem, i would love to hear it. I have played with a few plugins that do this sort of conversion but none that seem to play nice with camera. I'm considering writing one that does this native as suggested above... but... sigh... time... most likely, i will just write to a temp jpg file and read the bytes back until someone else does my job for me ;) — Rob C, Mar 26 '19 at 00:42
@AlexanderBlyth I'm having the same problem, using the stream and trying to get a base64 from each "frame". Did you get any solution? — Christian Benseler, Nov 27 '19 at 21:27

score 5 · Accepted Answer · answered Jun 13 '19 at 01:52

A solution to convert the image to a png:

Future<Image> convertYUV420toImageColor(CameraImage image) async {
  try {
    final int width = image.width;
    final int height = image.height;
    final int uvRowStride = image.planes[1].bytesPerRow;
    final int uvPixelStride = image.planes[1].bytesPerPixel;

    print("uvRowStride: " + uvRowStride.toString());
    print("uvPixelStride: " + uvPixelStride.toString());

    // imgLib -> Image package from https://pub.dartlang.org/packages/image
    var img = imglib.Image(width, height); // Create Image buffer

    // Fill image buffer with plane[0] from YUV420_888
    for(int x=0; x < width; x++) {
      for(int y=0; y < height; y++) {
        final int uvIndex = uvPixelStride * (x/2).floor() + uvRowStride*(y/2).floor();
        final int index = y * width + x;

        final yp = image.planes[0].bytes[index];
        final up = image.planes[1].bytes[uvIndex];
        final vp = image.planes[2].bytes[uvIndex];
        // Calculate pixel color
        int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
        int g = (yp - up * 46549 / 131072 + 44 -vp * 93604 / 131072 + 91).round().clamp(0, 255);
        int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);     
        // color: 0x FF  FF  FF  FF 
        //           A   B   G   R
        img.data[index] = (0xFF << 24) | (b << 16) | (g << 8) | r;
      }
    }

    imglib.PngEncoder pngEncoder = new imglib.PngEncoder(level: 0, filter: 0);
    List<int> png = pngEncoder.encodeImage(img);
    muteYUVProcessing = false;
    return Image.memory(png);  
  } catch (e) {
    print(">>>>>>>>>>>> ERROR:" + e.toString());
  }
  return null;
}

Source: https://github.com/flutter/flutter/issues/26348#issuecomment-462321428

How do you convert it for RBGA_8888 format (for iOS)? – Almas Adilbek Aug 13 '19 at 13:16 — Almas Adilbek, Aug 13 '19 at 13:16

score 0 · Answer 2 · edited Jan 22 '23 at 00:55

0

You can direct convert image file to base64 using this.

Image Encode:

var imageFilePath = await picker.getImage(source: ImageSource.gallery);

final bytes = ImageFilePath.readAsBytesSync();
String _img64 = base64Encode(bytes);

Image Decode:

_img64 = base64Decode(response.bodyBytes);
image.memory(_img64);

edited Jan 22 '23 at 00:55

Throvn

795
7
19

answered Feb 28 '21 at 16:41

Live Infosys

21
2

score 0 · Answer 3 · answered May 26 '21 at 15:59

Convert gallery image to base 64 Flutter

Future getImageFromGallery() async {
    var image = await ImagePicker.pickImage(source: ImageSource.gallery);
    final bytes = Io.File(image.path).readAsBytesSync();
    String img64 = base64Encode(bytes);
  }

score -1 · Answer 4 · answered Feb 28 '21 at 16:16

I am using this code to convert YUV_420 888 to PNG

// CameraImage YUV420_888 -> PNG -> Image (compresion:0, filter: none)
// Black
imglib.Image _convertYUV420(CameraImage image) {
  var img = imglib.Image(image.width, image.height); // Create Image buffer

  Plane plane = image.planes[0];
  const int shift = (0xFF << 24);

  // Fill image buffer with plane[0] from YUV420_888
  for (int x = 0; x < image.width; x++) {
    for (int planeOffset = 0;
    planeOffset < image.height * image.width;
    planeOffset += image.width) {
      final pixelColor = plane.bytes[planeOffset + x];
      // color: 0x FF  FF  FF  FF
      //           A   B   G   R
      // Calculate pixel color
      var newVal = shift | (pixelColor << 16) | (pixelColor << 8) | pixelColor;

      img.data[planeOffset + x] = newVal;
    }
  }

  return img;
}

And this to create a PNG

    Future<List<int>> convertImagetoPng(CameraImage image) async {
  try {
    imglib.Image img;
    if (image.format.group == ImageFormatGroup.yuv420) {
      img = convertYUV420_ToPNG(image);
    } else if (image.format.group == ImageFormatGroup.bgra8888) {
      img = convertBGRA8888_ToPNG(image);
    }

    imglib.PngEncoder pngEncoder = new imglib.PngEncoder();

    // Convert to png
    List<int> png = pngEncoder.encodeImage(img);
    return png;
  } catch (e) {
    print(">>>>>>>>>>>> ERROR:" + e.toString());
  }
  return null;
}

by hugand
You can check the performance as well

How to convert Flutter CameraImage to a Base64-encoded binary data object from an Image Stream

4 Answers4

Linked