How do I compute Laplacian in Tensorflow.js?

Question

I am currently trying to design a Node.js code that would be able to tell if an image is blurry or not. In order to achieve that, I took my inspiration from this question. So what I need to do is compute a matrix laplacian (is that how we say ?) and then compute variance.

I have no problem doing it with OpenCV (using opencv4nodejs) :

# load image
_cvImg = cv.imread _file

# get grayscale image
_cvImgGray =_cvImg.bgrToGray()

# Compute laplacian
_laplacian = _cvImgGray.laplacian(cv.CV_8U)

# Get the standard deviation
_meanStdDev = _laplacian.meanStdDev()

_stddevarray = _meanStdDev.stddev.getDataAsArray()
_stdDeviation = _stddevarray[0]

# Get the variation
_variation = Math.pow(_stdDeviation, 2)

But now, I am using Tensorflow.js and it is really less easy... Here what I tried to do:

# load image
_cvImg = cv.imread _file

#convert frame to a tensor
try
    _data = new Uint8Array(_frame.cvtColor(cv.COLOR_BGR2RGB).getData().buffer)
    _tensorFrame = tfjs.tensor3d(_data, [_frame.rows, _frame.cols, 3])
catch _err
    @log.error "Error instantiating tensor !!!"
    @log.error _err.message

# get grayscale image
_grayscaleFrame = _tensorFrame.mean(2).expandDims(2)

# prepare convolution to get laplacian
laplaceFilter = tfjs.tensor2d([[0,1,0], [1,-4,1], [0,1,0]])
laplaceFilter3D = laplaceFilter.expandDims(2)

# get laplacian
_laplacian = tfjs.conv1d _tensorFrame, laplaceFilter3D, 1, 'same'

# get standard deviation
_standardDeviation = tfjs.moments(_laplacian2).variance.buffer().values[0]

# get variance
_variance = _standardDeviation * _standardDeviation

# dispose tensor to avoid memeory leaks
_tensorFrame.dispose()

Not so surprisingly, the above code doesn't work. I know that my convolution should be two dimensional (tf.conv2d) instead of 1-D (tf.conv1d) since I am working on an image. If I look at tf.conv2d in API, I can see this signature:

tf.conv2d (x, filter, strides, pad, dataFormat?, dilations?, dimRoundingMode?)

And filter should be a tf.Tensor4D ! But I have no idea how the following filter matrix could be converted to tf.Tensor4D

    1
1   4   1   
    1

Am I doing something wrong ? How can I get laplacian of a matrix in Tensorflow.js ? i.e How can I perform a simple convolution between my tensor2d representing my image and the above filter ?

score 2 · Accepted Answer · answered Apr 09 '19 at 19:04

The convolution can be performed over the 2d tensor by first expanding its dimension. Likewise for the filter by using tf.reshape

After the computation, tf.squeeze will remove dimensions of shape 1, thus returning a 2d tensor

const x = tf.tensor2d(Array.from({length: 16}, (_, k) => k+1), [4, 4]); // input image 2d

const filter = tf.tensor2d([0, 1, 0, 1, 0, 1, 0, 1, 0], [3, 3]) // input filter 2d

x.reshape([1, ...x.shape, 1]).conv2d(filter.reshape([...filter.shape, 1, 1]), 1, 'valid').squeeze().print() // conv

<html>
  <head>
    <!-- Load TensorFlow.js -->
    <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@0.14.1"> </script>
  </head>

  <body>
  </body>
</html>

score 2 · Answer 2 · answered Jun 21 '20 at 09:03

Another solution:

// Reads the image, converts to gray and makes shape (1, width, height, 1) [batch, in_height, in_width, in_channels]
let _grayscaleFrame = tf.browser.fromPixels(canvas).mean(2).toFloat().expandDims(0).expandDims(-1); 
                                                          

// prepare convolution to get laplacian
let laplaceFilter = tf.tensor2d([
          [0, 1, 0],
          [1, -4, 1],
          [0, 1, 0]
]).expandDims(-1).expandDims(-1); // [filter_height, filter_width, in_channels, out_channels]

// get laplacian
let _laplacian = _grayscaleFrame.conv2d(laplaceFilter, 1, 'valid').squeeze()

How do I compute Laplacian in Tensorflow.js?

2 Answers2