9

I was curious of the new "turn on/off" background blur functionality of Google Meet (currently in test). I have investigated a bit and it seems it is using Tensorflow Lite models:

segm_heavy.tflite
segm_lite.tflite

via WASM

mediapipe_wasm_simd.wasm

while the model graph should be

background_blur_graph.binarypb

The model seems works at the level of the HTMLCanvasElement as far as I can see. Anyone aware of a similar model?

[UPDATE]

Thanks to Jason Mayes and Physical Ed I was able to reproduce a very close background blur effect in the Google's BodyPix demo

The settings of the application are showed in the Controls box. There is a backgroundBlurAmount option that let you customize the blur percentage to apply as well.

enter image description here

The result is almost close to the official Google Meet application.

enter image description here

loretoparisi
  • 15,724
  • 11
  • 102
  • 146
  • 2
    Our Bodypix model can allow you to segment the human body and blur the background. https://github.com/tensorflow/tfjs-models/tree/master/body-pix – Jason Mayes Oct 01 '20 at 01:14
  • @JasonMayes thanks. I was looking at the Google’s Media Pipe that seems to use TF lite and have human body tracking. Is this the sane model you are referring to? – loretoparisi Oct 01 '20 at 08:13
  • 1
    @loretoparisi I can answer that, we are experimenting with using body-pix. It is a different model with different capabilities than mediapipe. It has an excellent interface and a very good demo. Check it out by all means. – PhysicalEd Oct 02 '20 at 21:32
  • 1
    Indeed as PysicalEd said Bodypix is different. You can see demo of it here: https://github.com/tensorflow/tfjs-models/tree/master/body-pix – Jason Mayes Oct 05 '20 at 04:12
  • @loretoparisi, Is your issue resolved? If so, can you please post the solution in answer section for the benefit of the Community. Thanks! –  Nov 06 '20 at 08:27
  • @TFer2 thank you for the reminder. I will do with more details. – loretoparisi Nov 06 '20 at 09:15

1 Answers1

3

majority of segmentation models give alpha channel as a result (some give more, but alpha is most useful) - what is masked and what is not

so if you want to blur the background, its a multi-step process:

  1. resize input to model expected size
  2. run model to get alpha channel
  3. resize output back to original size
  4. draw original image on canvas
  5. draw alpha channel over it so only foreground stays visible
    for example using ctx.globalCompositeOperation = 'darken'
  6. optionally blur it a bit since model output is never perfect
    for example using ctx.filter = blur(8px)`;

so if you want to blur the background, simply apply apply blur simply copy canvas from 4, apply blur on it and draw if back before going to step 5

regarding models, google meet is not bad but i had better results with google selfie model
bodypix is older model, great configurability but not that great results

example code: https://github.com/vladmandic/human/blob/main/src/segmentation/segmentation.ts

Vladimir Mandic
  • 813
  • 5
  • 11