Barcode Google Vision API check if detected barcode is inside area

Question

I have problem with detecting if barcode is inside specified area. For testing purposes camera source preview and surface view has same size 1440x1080 to prevent scaling between camera and view. I get positive checks even if I see QR Code isn't in box what represents image. Whats wrong?

False positive check

ScannerActivity

public class ScannerActivity extends AppCompatActivity {
    private static final String TAG = "ScannerActivity";

    private SurfaceView mSurfaceView; // Its size is forced to 1440x1080 in XML
    private CameraSource mCameraSource;
    private ScannerOverlay mScannerOverlay; // Its size is forced to 1440x1080 in XML

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        // .. create and init views
        // ...

        BarcodeDetector barcodeDetector = new BarcodeDetector.Builder(this)
                .setBarcodeFormats(Barcode.ALL_FORMATS)
                .build();

        mCameraSource = new CameraSource.Builder(this, barcodeDetector)
                .setRequestedPreviewSize(1440, 1080)
                .setRequestedFps(20.0f)
                .setFacing(CameraSource.CAMERA_FACING_BACK)
                .setAutoFocusEnabled(true)
                .build();

        barcodeDetector.setProcessor(new Detector.Processor<Barcode>() {
            @Override
            public void release() {
            }

            @Override
            public void receiveDetections(Detector.Detections<Barcode> detections) {
                parseDetections(detections.getDetectedItems());
            }
        });

    }

    private void parseDetections(SparseArray<Barcode> barcodes) {
        for (int i = 0; i < barcodes.size(); i++) {
            Barcode barcode = barcodes.valueAt(i);
            if (isInsideBox(barcode)) {
                runOnUiThread(() -> {
                    Toast.makeText(this, "GOT DETECTION: " + barcode.displayValue, Toast.LENGTH_SHORT).show();
                });
            }
        }
    }

    private boolean isInsideBox(Barcode barcode) {
        Rect barcodeBoundingBox = barcode.getBoundingBox();
        Rect scanBoundingBox = mScannerOverlay.getBox();

        boolean checkResult = barcodeBoundingBox.left >= scanBoundingBox.left &&
                barcodeBoundingBox.right <= scanBoundingBox.right &&
                barcodeBoundingBox.top >= scanBoundingBox.top &&
                barcodeBoundingBox.bottom <= scanBoundingBox.bottom;

        Log.d(TAG, "isInsideBox: "+(checkResult ? "YES" : "NO"));
        return checkResult;
    }
}

score 1 · Answer 1 · answered Jul 22 '20 at 12:07

1

Explanation to your issue is simple, but the solution is not trivial to explain.

The coordinates of the box from your UI will mostly not the be the same like the imaginary box on each preview frame. You must transform the coordinates from the UI box to scanBoundingBox.

I open sourced an example which implement the same usecase you are trying to accomplish. In this example I took another approach, I cut the box out of each frame first before feeding it to Google Vision, which is also more efficient, since Google Vision don't have to analyse the whole picture and waste tons of CPU...

answered Jul 22 '20 at 12:07

Minki

934
5
14

Sounds strange but based on your example I will try. Did you hear about ``FocussingProcessor``? I read that it can be used for filtering. – Remzo Jul 22 '20 at 12:25
just took a look of it, but this is not the thing you want, what you want is to crop the box of each frame and check if a QR code is recognised within the box – Minki Jul 22 '20 at 12:29
Of course, you are right but I'm wondering if there is any other, easier option to achieve same result - check if barcode is in box. As I can see on your example project you have spent a lot of time with Google Vision so I'm asking you as expert if using approach from this post https://stackoverflow.com/a/36428822/7674399 is it possible to do what I want? Or there will be still errors with comparing valid coordinates? – Remzo Jul 22 '20 at 13:12
1

Sure you can also use this approach, you only need to get the coordinates transformation right. It is just inefficient if you feed the detector the whole image instead of only part of it, thats why I think the cropping approach is the better practise. – Minki Jul 22 '20 at 13:19
Thank you for your answers, if i would have more questions about this problem I will ask here. – Remzo Jul 22 '20 at 13:46

score 0 · Answer 2 · answered Jul 23 '20 at 10:54

I decied to cropp frame by wrapping barcode detector however I don't know why but cropped frame is rotated by 90 degress even smarphone is upright orientation.

Box Detector class

public class BoxDetector extends Detector<Barcode> {

    private Detector<Barcode> mDelegate;
    private int mBoxWidth;
    private int mBoxHeight;

    // Debugging
    private CroppedFrameListener mCroppedFrameListener;

    public BoxDetector(Detector<Barcode> delegate, int boxWidth, int boxHeight) {
        mDelegate = delegate;
        mBoxWidth = boxWidth;
        mBoxHeight = boxHeight;
    }

    public void setCroppedFrameListener(CroppedFrameListener croppedFrameListener) {
        mCroppedFrameListener = croppedFrameListener;
    }

    @Override
    public SparseArray<Barcode> detect(Frame frame) {
        int frameWidth = frame.getMetadata().getWidth();
        int frameHeight = frame.getMetadata().getHeight();

        // I assume that box is centered.
        int left = (frameWidth / 2) - (mBoxWidth / 2);
        int top = (frameHeight / 2) - (mBoxHeight / 2);
        int right = (frameWidth / 2) + (mBoxWidth / 2);
        int bottom = (frameHeight / 2) + (mBoxHeight / 2);

        YuvImage yuvImage = new YuvImage(frame.getGrayscaleImageData().array(), ImageFormat.NV21, frameWidth, frameHeight, null);
        ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
        yuvImage.compressToJpeg(new Rect(left, top, right, bottom), 100, outputStream);
        byte[] jpegArray = outputStream.toByteArray();
        Bitmap bitmap = BitmapFactory.decodeByteArray(jpegArray, 0, jpegArray.length);

        Frame croppedFrame = new Frame.Builder()
                .setBitmap(bitmap)
                .setRotation(frame.getMetadata().getRotation())
                .build();

        if(mCroppedFrameListener != null) {
            mCroppedFrameListener.onNewCroppedFrame(croppedFrame.getBitmap(), croppedFrame.getMetadata().getRotation());
        }

        return mDelegate.detect(croppedFrame);
    }

    public interface CroppedFrameListener {
        void onNewCroppedFrame(Bitmap bitmap, int rotation);
    }
}

Box Detector usuage

        BarcodeDetector barcodeDetector = new BarcodeDetector.Builder(this)
                .setBarcodeFormats(Barcode.ALL_FORMATS)
                .build();

        BoxDetector boxDetector = new BoxDetector(
                barcodeDetector,
                mBoxSize.getWidth(),
                mBoxSize.getHeight());

        boxDetector.setCroppedFrameListener(new BoxDetector.CroppedFrameListener() {
            @Override
            public void onNewCroppedFrame(final Bitmap bitmap, int rotation) {
                Log.d(TAG, "onNewCroppedFrame: new bitmap, rotation: "+rotation);
                runOnUiThread(new Runnable() {
                    @Override
                    public void run() {
                        mPreview.setImageBitmap(bitmap);
                    }
                });
            }
        });

Barcode Google Vision API check if detected barcode is inside area

False positive check

ScannerActivity

2 Answers2

Box Detector class

Box Detector usuage

Cropped frame is rotated