CameraX Image analysis's imageproxy size and PreviewView size are not the same

Question

I'm trying to use Firebase's MLKit for face detection with Camerax. I'm having a hard time to get Image analysis's imageproxy size to match PreviewView's size. For both Image analysis and PreviewView, I've set setTargetResolution() to PreviewView width and height. However when I check the size of the Imageproxy in the analyzer, it's giving me 1920 as width and 1080 as height. My PreviewView is 1080 for width and 2042 for height. When I swap the width and the height in setTargetResolution() for Image analysis, I get 1088 for both width and height in imageproxy. My previewview is also locked to portrait mode.

Ultimately, I need to feed the raw imageproxy data and the face point data into an AR code. So scaling up just the graphics overlay that draws the face points will not work for me.

Q: If there are no way to fix this within the camerax libraries, How to scale the imageproxy that returns from the analyzer to match the previewview?

I'm using Java and the latest Camerax libs:

def camerax_version = "1.0.0-beta08"

Answer 1

It's quite difficult to ensure both the preview and image analysis use cases have the same output resolution, since different devices support different resolutions, and image analysis has a hard limit on the max resolution of its output (as mentioned in the documentation).

To make the conversion easier between coordinates from the image analysis frames and the UI/PreviewView, you can set both preview and ImageAnalysis to use the same aspect ratio, for instance AspectRatio.RATIO_4_3, as well as PreviewView (by wrapping it inside a ConstraintLayout for example, and setting a constraint on its width/height ratio). With this, mapping coordinates of detected faces from the analyzer to the UI becomes more straight-forward, you can take a look at it in this sample.

Alternatively, you could use CameraX's ViewPort API which -I believe- is still experimental. It allows defining a field of view for a group of use cases, resulting in their outputs matching and having WYSIWYG. You can find an example of its usage here. For your case, you'd write something like this.

Preview preview = ...
preview.setSurfaceProvider(previewView.getSurfaceProvider());

ImageAnalysis imageAnalysis = ...
imageAnalysis.setAnalyzer(...);

ViewPort viewPort = preview.getViewPort();
UseCaseGroup useCaseGroup = new UseCaseGroup.Builder()
                .setViewPort(viewPort)
                .addUseCase(preview)
                .addUseCase(imageAnalysis)
                .build();

cameraProvider.bindToLifecycle(
                lifecycleOwner,
                cameraSelector,
                usecaseGroup);

In this scenario, every ImageProxy your analyzer receives will contain a crop rect that matches what PreviewView displays. So you just need to crop your image, then pass it to the face detector.

Answer 2

This answer is derived from @Husayn's answer. I have added relevant sample code part.

Camerax image size for preview and analysis varies for various reasons (example device specific display size/hardware/camera or app specific view and processing) However there are options to map the processing image size and resulting xy coordinates to preview size and to preview xy coordinates.

Setup layout with DimensionRatio 3:4 for both preview and analysis overlay in layout,

Example:

<androidx.camera.view.PreviewView
  android:id="@+id/view_finder"
  android:layout_width="match_parent"
  android:layout_height="0dp"
  app:layout_constraintBottom_toBottomOf="parent"
  app:layout_constraintDimensionRatio="3:4"
  app:layout_constraintTop_toTopOf="parent"/>

<com.loa.sepanex.scanner.view.GraphicOverlay
  android:id="@+id/graphic_overlay"
  android:layout_width="match_parent"
  android:layout_height="0dp"
  app:layout_constraintBottom_toBottomOf="parent"
  app:layout_constraintDimensionRatio="3:4"
  app:layout_constraintTop_toTopOf="parent"/>

setup preview and analysis use cases wuth AspectRatio.RATIO_4_3

Example:

viewFinder = view.findViewById(R.id.view_finder)
graphicOverlay = view.findViewById(R.id.graphic_overlay)
//...
preview = Preview.Builder()
          .setTargetAspectRatio(AspectRatio.RATIO_4_3)
          .setTargetRotation(rotation)
          .build()

imageAnalyzer = ImageAnalysis.Builder()
                .setTargetAspectRatio(AspectRatio.RATIO_4_3)
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
                .setTargetRotation(rotation)
                .build()
                .also {
                    it.setAnalyzer(cameraExecutor, ImageAnalysis.Analyzer { 
                     image ->
                        //val rotationDegrees = image.imageInfo.rotationDegrees
                        try {
                            val mediaImage: Image? = image.image
                            if (mediaImage != null) {
                                val imageForFaceDetectionProcess = InputImage.fromMediaImage(mediaImage, image.getImageInfo().getRotationDegrees())
                                //...
                            }
                         }
                      }
                  }

Define scale and traslate APIs for getting the mapping of analysis image xy coordinates to preview xy coordinates, as shown below

            val preview = viewFinder.getChildAt(0)
            var previewWidth = preview.width * preview.scaleX
            var previewHeight = preview.height * preview.scaleY
            val rotation = preview.display.rotation
            if (rotation == Surface.ROTATION_90 || rotation == Surface.ROTATION_270) {
                val temp = previewWidth
                previewWidth = previewHeight
                previewHeight = temp
            }
            val isImageFlipped = lensFacing == CameraSelector.LENS_FACING_FRONT
            val rotationDegrees: Int = imageProxy.getImageInfo().getRotationDegrees()
            if (rotationDegrees == 0 || rotationDegrees == 180) {
                graphicOverlay!!.setImageSourceInfo(
                        imageProxy.getWidth(), imageProxy.getHeight(), isImageFlipped)
            } else {
                graphicOverlay!!.setImageSourceInfo(
                        imageProxy.getHeight(), imageProxy.getWidth(), isImageFlipped)
            }
    :::
    :::
    float viewAspectRatio = (float) previewWidth / previewHeight;
    float imageAspectRatio = (float) imageWidth / imageHeight;
    postScaleWidthOffset = 0;
    postScaleHeightOffset = 0;

    if (viewAspectRatio > imageAspectRatio) {
        // The image needs to be vertically cropped to be displayed in this view.
        scaleFactor = (float) previewWidth / imageWidth;
        postScaleHeightOffset = ((float) previewWidth / imageAspectRatio - previewHeight) / 2;
    } else {
        // The image needs to be horizontally cropped to be displayed in this view.
        scaleFactor = (float) previewHeight / imageHeight;
        postScaleWidthOffset = ((float) previewHeight * imageAspectRatio - previewWidth) / 2;
    }
    transformationMatrix.reset();
    transformationMatrix.setScale(scaleFactor, scaleFactor);
    transformationMatrix.postTranslate(-postScaleWidthOffset, -postScaleHeightOffset);
    if (isImageFlipped) {
        transformationMatrix.postScale(-1f, 1f, previewWidth / 2f, previewHeight / 2f);
    }
    :::
    :::
    public float scale(float imagePixel) {
        return imagePixel * overlay.scaleFactor;
    }
    public float translateX(float x) {
        if (overlay.isImageFlipped) {
            return overlay.getWidth() - (scale(x) - overlay.postScaleWidthOffset);
        } else {
            return scale(x) - overlay.postScaleWidthOffset;
        }
    }
    public float translateY(float y) {
        return scale(y) - overlay.postScaleHeightOffset;
    }

use translateX and translateY methods for plotting analysis image based data into preview

Example:

        for (FaceContour contour : face.getAllContours()) {
            for (PointF point : contour.getPoints()) {
                canvas.drawCircle(translateX(point.x), translateY(point.y), FACE_POSITION_RADIUS, facePositionPaint);
            }
        }