Effective blurring of camera preview

Question

What I've tried so far:

---

Convert every frame into bitmap, blur it with library and put it into ImageView which is in front of camera preview. Obviously was too slow - something like 1 fps.

---

Then I started to use RenderScript which blurs every frame and result of processing should be placed in TextureView which is cover camera preview.

Essential peaces of code of that approach:

BlurFilter

ScriptIntrinsicBlur.create(rs, Element.RGBA_8888(rs)).apply {
    setRadius(BLUR_RADIUS)
}
private val yuvToRgb = ScriptIntrinsicYuvToRGB.create(rs, Element.RGBA_8888(rs))
private var surface: SurfaceTexture? = null

private fun setupSurface() {
    if (surface != null) {
        aBlurOut?.surface = Surface(surface)
    }
}

fun reset(width: Int, height: Int) {
    aBlurOut?.destroy()

    this.width = width
    this.height = height

    val tbConvIn = Type.Builder(rs, Element.U8(rs))
            .setX(width)
            .setY(height)
            .setYuvFormat(android.graphics.ImageFormat.NV21)
    aConvIn = Allocation.createTyped(rs, tbConvIn.create(), Allocation.USAGE_SCRIPT)

    val tbConvOut = Type.Builder(rs, Element.RGBA_8888(rs))
            .setX(width)
            .setY(height)
    aConvOut = Allocation.createTyped(rs, tbConvOut.create(), Allocation.USAGE_SCRIPT)

    val tbBlurOut = Type.Builder(rs, Element.RGBA_8888(rs))
            .setX(width)
            .setY(height)
    aBlurOut = Allocation.createTyped(rs, tbBlurOut.create(),
            Allocation.USAGE_SCRIPT or Allocation.USAGE_IO_OUTPUT)

    setupSurface()
}

fun execute(yuv: ByteArray) {
    if (surface != null) {
        //YUV -> RGB
        aConvIn!!.copyFrom(yuv)
        yuvToRgb.setInput(aConvIn)
        yuvToRgb.forEach(aConvOut)
        //RGB -> BLURED RGB
        blurRc.setInput(aConvOut)
        blurRc.forEach(aBlurOut)
        aBlurOut!!.ioSend()
    }
}

MainActivity

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_main)

    initQrScanner()
}

override fun onStart() {
    super.onStart()
    fotoapparat.start()
}

override fun onStop() {
    fotoapparat.stop()
    super.onStop()
}

private fun initQrScanner() {
    val filter = BlurFilter(RenderScript.create(this))
    tvWholeOverlay.surfaceTextureListener = filter

    fotoapparat = Fotoapparat
            .with(this)
            .into(cvQrScanner)
            .frameProcessor({
                if (it.size.width != filter.width || it.size.height != filter.height) {
                    filter.reset(it.size.width, it.size.height)
                }
                filter.execute(it.image)
            })
            .build()
}

activity_main.xml

<android.support.constraint.ConstraintLayout 
xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context="com.blur.andrey.blurtest.MainActivity">

    <io.fotoapparat.view.CameraView
        android:id="@+id/cvQrScanner"
        android:layout_width="match_parent"
        android:layout_height="match_parent" />

    <TextureView
        android:id="@+id/tvWholeOverlay"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        app:layout_constraintTop_toTopOf="parent" />

</android.support.constraint.ConstraintLayout>

And unfortunately it is still to slow - 3-4 FPS. Also blurring overlay is rotated, but it's another problem.

---

I've created test project on Github where you can quickly reproduce problem and check how it is possible to optimise. Looking forward for your ideas.

---

UPD I was able to improve performance with scaling down input date before blurring. I pushed those changes to test repo. Now I have really good (15-20 FPS) performance even on low end devices, but with low res (HD for instance), and not good enough on FHD and UHD ((8-12 FPS).

Answer 1

I think the best approach you can take is to replace CameraView with a TextureView and use it as a camera preview. You can easily find examples how to do it here https://developer.android.com/reference/android/view/TextureView.html, and here https://github.com/dalinaum/TextureViewDemo/blob/master/src/kr/gdg/android/textureview/CameraActivity.java.

Next step is to use a blurring shader on your TextureView. You can find an example here: https://github.com/nekocode/blur-using-textureview/blob/master/app/src/main/java/cn/nekocode/blurringview/sample/BlurFilter.java

You can use this shader or find some more nice-looking one.

In case if you want a ready to use solution, you can take a look at this library: https://github.com/krazykira/VidEffects

It allows you to apply shaders to a Video View, so again, you can use a blurring shader to achieve a desired effect.

Example for this library: https://stackoverflow.com/a/38125734/3286059

EDIT

So I forked a great library from Nekocode and added a working blur filter. Please see my latest commit. https://github.com/Dimezis/CameraFilter

As I said, TextureView + GL shaders. To enable blur shader, click on menu in top right corner and select the corresponding option.

If you need faster/better/simpler blur shader, you can search for one on https://www.shadertoy.com

On some devices you might camera preview flipped, but that's another problem to solve.

Answer 2

I have achieved nice live camera blur effect with RenderScript and previewing the output result to the ImageView.

1. First I created a BlurBuilder class which uses renderscript

    public class BlurBuilder {
    private static final float BITMAP_SCALE = 4f;
    private static final float BLUR_RADIUS = 25f; // 0 - 25
   
    public static Bitmap blur(Context context, Bitmap image) {
        int width = Math.round(image.getWidth() * BITMAP_SCALE);
        int height = Math.round(image.getHeight() * BITMAP_SCALE);
        Bitmap inputBitmap = Bitmap.createScaledBitmap(image, width, height, false);
        Bitmap outputBitmap = Bitmap.createBitmap(inputBitmap);
        RenderScript rs = RenderScript.create(context);
        ScriptIntrinsicBlur intrinsicBlur = ScriptIntrinsicBlur.create(rs, Element.U8_4(rs));
        Allocation tmpIn = Allocation.createFromBitmap(rs, inputBitmap);
        Allocation tmpOut = Allocation.createFromBitmap(rs, outputBitmap);
        intrinsicBlur.setRadius(BLUR_RADIUS);
        intrinsicBlur.setInput(tmpIn);
        intrinsicBlur.forEach(tmpOut);
        tmpOut.copyTo(outputBitmap);
        return outputBitmap;
    }}
   

2. I used TextureView to preview the camera preview

3. Add a ImageView over the TextureView

4. Add a setSurfaceTextureListener and I added a little tweak

@Override
    public void onSurfaceTextureUpdated(SurfaceTexture surface) {
        Bitmap bm = BlurBuilder.blur(getApplicationContext(),textureView.getBitmap());
        imageView.setImageBitmap(bm);
    }

PS: I used Opengles in TextureView.

My Result: https://vimeo.com/517761912 Device Name: Samsung j7 prime OS Version: Android 8.1 Camera: 13mp 30fps (Device capacity)