InvalidArgumentError: cannot compute MatMul as input #0(zero-based) was expected to be a float tensor but is a double tensor [Op:MatMul]

Question

Can somebody explain, how does TensorFlow's eager mode work? I am trying to build a simple regression as follows:

import tensorflow as tf
import numpy as np

tfe = tf.contrib.eager
tf.enable_eager_execution()

def make_model():
    net = tf.keras.Sequential()
    net.add(tf.keras.layers.Dense(4, activation='relu'))
    net.add(tf.keras.layers.Dense(1))
    return net

def compute_loss(pred, actual):
    return tf.reduce_mean(tf.square(tf.subtract(pred, actual)))

def compute_gradient(model, pred, actual):
    """compute gradients with given noise and input"""
    with tf.GradientTape() as tape:
        loss = compute_loss(pred, actual)
    grads = tape.gradient(loss, model.variables)
    return grads, loss

def apply_gradients(optimizer, grads, model_vars):
    optimizer.apply_gradients(zip(grads, model_vars))
    
model = make_model()
optimizer = tf.train.AdamOptimizer(1e-4)

x = np.linspace(0,1,1000)
y = x + np.random.normal(0,0.3,1000)
y = y.astype('float32')
train_dataset = tf.data.Dataset.from_tensor_slices((y.reshape(-1,1)))

epochs = 2# 10
batch_size = 25
itr = y.shape[0] # batch_size
for epoch in range(epochs):
    for data in tf.contrib.eager.Iterator(train_dataset.batch(25)):
        preds = model(data)
        grads, loss = compute_gradient(model, preds, data)
        apply_gradients(optimizer, grads, model.variables)
# Gradient output: [None, None, None, None, None, None]

The error is following:

----------------------------------------------------------------------
ValueError                           Traceback (most recent call last)
<ipython-input-3-a589b9123c80> in <module>
     35         grads, loss = compute_gradient(model, preds, data)
     36         print(grads)
---> 37         apply_gradients(optimizer, grads, model.variables)
     38 #         with tf.GradientTape() as tape:
     39 #             loss = tf.sqrt(tf.reduce_mean(tf.square(tf.subtract(preds, data))))

<ipython-input-3-a589b9123c80> in apply_gradients(optimizer, grads, model_vars)
     17 
     18 def apply_gradients(optimizer, grads, model_vars):
---> 19     optimizer.apply_gradients(zip(grads, model_vars))
     20 
     21 model = make_model()

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py in apply_gradients(self, grads_and_vars, global_step, name)
    589     if not var_list:
    590       raise ValueError("No gradients provided for any variable: %s." %
--> 591                        ([str(v) for _, v, _ in converted_grads_and_vars],))
    592     with ops.init_scope():
    593       self._create_slots(var_list)

ValueError: No gradients provided for any variable:

Edit

I updated my code. Now, the problem comes in gradients calculation, it is returning zero. I have checked the loss value that is non-zero.

Anna Krogager · Accepted Answer · 2019-01-19T10:22:50.970

47

Part 1: The problem is indeed the datatype of your input. By default your keras model expects float32 but you are passing a float64. You can either change the dtype of the model or change the input to float32.

To change your model:

def make_model():
    net = tf.keras.Sequential()
    net.add(tf.keras.layers.Dense(4, activation='relu', dtype='float32'))
    net.add(tf.keras.layers.Dense(4, activation='relu'))
    net.add(tf.keras.layers.Dense(1))
    return net

To change your input: y = y.astype('float32')

Part 2: You need to call the function that computes your model (i.e. model(data)) under tf.GradientTape(). For example, you can replace your compute_loss method with the following:

def compute_loss(model, x, y):
    pred = model(x)
    return tf.reduce_mean(tf.square(tf.subtract(pred, y)))

edited Jan 19 '19 at 10:22

answered Jan 18 '19 at 14:16

Anna Krogager

3,528
16
23

1

Thanks @Anna, it is working now. Now, new problem arise. my loss is ``, but when i compute gradient, it return `None` for all variable. I use `GradientTape` to calculate it as `tape.gradient(loss, model.variables)`. – Ankish Bansal Jan 18 '19 at 15:05
1

Can you provide the full code that you are using for this? – Anna Krogager Jan 18 '19 at 16:25
1

I edited my answer so that it answers both of your questions. Hope this helps! – Anna Krogager Jan 19 '19 at 10:23

score 0 · Answer 2 · answered Jun 22 '23 at 21:03

If you arrived at this Q/A because you got the error in the title and your input is an EagerTensor, then you can cast one of the matrices in the matrix multiplication to the appropriate dtype using tf.cast().

For example, in the following example, x is of dtype float while y is of dtype double (float64 is double), so trying to perform matrix multiplication as is would throw an InvalidArgumentError: cannot compute MatMul as input #1(zero-based) was expected to be a float tensor but is a double tensor [Op:MatMul]. To solve the error, cast one of the matrices to the dtype of the other.

import tensorflow as tf

x = tf.constant([[1, 2]], dtype='float32')
y = tf.constant([[3], [4]], dtype='float64')

tf.matmul(x, y)                         # <---- InvalidArgumentError
z = tf.matmul(x, tf.cast(y, x.dtype))   # <----- OK
z.dtype   # tf.float32

w = tf.matmul(tf.cast(x, y.dtype), y)   # <----- OK
w.dtype   # tf.float64

InvalidArgumentError: cannot compute MatMul as input #0(zero-based) was expected to be a float tensor but is a double tensor [Op:MatMul]

Edit

2 Answers2

Linked