I get the following error:
2020-03-08 17:48:56.662834: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2020-03-08 17:48:56.662905: W tensorflow/stream_executor/stream.cc:2041] attempting to perform BLAS operation using StreamExecutor without BLAS support
2020-03-08 17:48:56.663040: I tensorflow/stream_executor/stream.cc:4976] [stream=0x185fdc10,impl=0x18379c60] did not memset GPU location; source: 0x7f3c66ffa2c0; size: 8388608; pattern: ffffffff
2020-03-08 17:48:56.663136: I tensorflow/stream_executor/stream.cc:4976] [stream=0x185fdc10,impl=0x18379c60] did not memset GPU location; source: 0x7f3c66ffa320; size: 8388608; pattern: ffffffff
2020-03-08 17:48:56.664750: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Internal: BlasScal failed : in.shape=[2,256,256]
[[{{node loss/p_re_lu_15_loss/lambda_1/IFFT2D}}]]
2020-03-08 17:48:56.689041: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2/16 [==>...........................] - ETA: 28sTraceback (most recent call last):
the network compiles with the loss function, it is only at execution that it fails.
my loss function is:
def my_custom_loss_fun(y_true, y_pred):
y_pred_complex_cast=tf.cast(y_pred,tf.complex64)
y_true_complex_cast=tf.cast(y_true,tf.complex64)
y_pred_complex=tf.keras.layers.Lambda(tf.signal.ifft2d)(tf.math.add(y_pred_complex_cast[:,:,:,0],tf.math.multiply(tf.cast(1j,tf.complex64),y_pred_complex_cast[:,:,:,0])))
y_true_complex=tf.keras.layers.Lambda(tf.signal.ifft2d)(tf.math.add(y_true_complex_cast[:,:,:,1],tf.math.multiply(tf.cast(1j,tf.complex64),y_true_complex_cast[:,:,:,1])))
return tf.losses.MSE(tf.abs(y_pred_complex),tf.abs(y_true_complex))
The CNN must take as input matrices of shape (256,256,2). Any ideas?
my imports are:
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Dense, Flatten, Conv1D, Conv2D, PReLU, Dropout, Input, Dropout, concatenate, MaxPooling2D, Activation, ReLU, MaxPooling2D
from tensorflow.keras.layers import LeakyReLU, Softmax, Cropping2D, UpSampling2D#,regularizers
from tensorflow.keras import initializers as initializers
import tensorflow as tf
Also the loss fun seems to return a vector not a scalar which is strange since I expect MSE to return a scalar of course...
I added reduce_mean to get a scalar ---> but same error.
Also I have:
failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
do I need to initialize something?
