Custom layer with Keras and Tensorflow

Question

I am building a custom layer using Keras and Tensorflow, that calculate the intersection over union between a set of boxes. a have two sets anchors and gt_boxes with different dmensions, and I will calculate the intersection over union between each element in anchors and gt_boxes when I execute the code the following error:

Use fn_output_signature instead
Traceback (most recent call last):
  File "acceuil.py", line 60, in <module>
    train(train_data[1], val_data[1], dataset_name)
  File "acceuil.py", line 28, in train
    model.train()
  File "/home/imene/APP-M/ROI/mod.py", line 40, in train
    inputs, outputs =   self.worker() 
  File "/home/imene/APP-M/ROI/mod.py", line 135, in worker
    iou_anchors = ioulayer([tf_rois, tf_anchors])
  File "/home/imene/anaconda3/envs/master/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 786, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/home/imene/anaconda3/envs/master/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 670, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:

    /home/imene/APP-M/ROI/IoULayer.py:25 call  *
        IoU_anchors = tf.map_fn(compute_IoU, (inputs[0], inputs[1] ), dtype=tf.float32)
    /home/imene/anaconda3/envs/master/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py:538 new_func  **
        return func(*args, **kwargs)
    /home/imene/anaconda3/envs/master/lib/python3.6/site-packages/tensorflow/python/ops/map_fn.py:451 map_fn
        tensor.get_shape().with_rank_at_least(1)[0])))
    /home/imene/anaconda3/envs/master/lib/python3.6/site-packages/tensorflow/python/framework/tensor_shape.py:315 merge_with
        self.assert_is_compatible_with(other)
    /home/imene/anaconda3/envs/master/lib/python3.6/site-packages/tensorflow/python/framework/tensor_shape.py:282 assert_is_compatible_with
        (self, other))

    ValueError: Dimensions 3 and 10 are not compatible

here is the code of my custom layer, written with python:

import sys
import numpy as np 
import keras
from tensorflow.keras.layers import Layer
from tensorflow.python.keras.backend import map_fn

import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

class IoULayer(Layer):
    def __init__(self, **kwargs):
        super(IoULayer, self).__init__(**kwargs)
        
        
# -------------------------------------------------------------------------------------------------
    def call(self, inputs):
        '''
        inputs[0]: ground truth boxes
        inputs[1]: anchors boxes
        '''    
        def compute_IoU(inputs):
            
            return IoULayer.multiple_IoU(inputs[0], inputs[1])

        IoU_anchors = tf.map_fn(compute_IoU, (inputs[0], inputs[1] ), dtype=tf.float32)
        
        return IoU_anchors

    def multiple_IoU(gt_boxes, anchors):

        def get_anchor_IoUs(gt_boxes, anchor):
            
            return IoULayer.get_single_IoU(gt_boxes, anchor)

        IoU = tf.map_fn(get_anchor_IoUs, gt_boxes, anchors, dtype=tf.float32)
        return IoU

    def get_single_IoU(gt_boxes, anchors):
        iou_list = []
        
        def single_iou(anchor):
            result = []
            for gt_bbx in gt_boxes:
                
                x_left   = max(gt_bbx[0], anchor[0])
                y_top    = max(gt_bbx[1], anchor[1])
                x_right  = min(gt_bbx[2], anchor[2])
                y_bottom = min(gt_bbx[3], anchor[3])

                bb1_area = (gt_bbx[2]- gt_bbx[0])*(gt_bbx[3]- gt_bbx[1])
                anchor_area = (anchor[2]- anchor[0])*(anchor[3]- anchor[1])

                intersect_area = abs(max((x_right - x_left), 0) * max((y_bottom - y_top),0))
                iou = intersect_area / float(bb1_area + anchor_area - intersect_area)
                result.append(iou)
                return result
        iou_list = tf.stack(single_iou(anchor) for anchor in anchors)
        return iou_list

I don't understand what is the problem? Thanks for replying

Answer 1

The problem might be in the dimensions of your input. From tf.map_fn documentation:

If elems is a tuple (or nested structure) of tensors, then those tensors must all have the same outer-dimension size (num_elems); and fn is used to transform each tuple (or structure) of corresponding slices from elems. E.g., if elems is a tuple (t1, t2, t3), then fn is used to transform each tuple of slices (t1[i], t2[i], t3[i]) (where 0 <= i < num_elems).

So in your case if tf_rois and tf_anchors have different dimension (like 3 and 10), fn_map will fail because there are not enough elements in the smaller tensor to pair with the bigger tensor elements.

If you need to run your functions against all permutations of the two tensors, you need to find those combinations and feed two equal-sized tensors as elems in fn_map.

EDIT: more detail on the combinations.

Instead of feeding the two raw tensors to tf.map_fn, you can find all the pairwise combinations, so if your tensors look like this:

tensor_1 = [1, 2, 3]
tensor_2 = [100, 200, 300]

The tensor of pairwise combinations will look like this:

pairwise_combinations = [
    [1, 100],
    [1, 200],
    [1, 300], 
    [2, 100],
    [2, 200],
    [2, 300],
    [3, 100],
    [3, 200],
    [3, 300]
]

Now you can feed columns of this tensor as equal-sized elems to the map_fn.

About the actual way you can find those combinations, here is a link with a tensorflow example and here is how it can be done in python standard library. Hope it helps!