How to solve "No Algorithm Worked" Keras Error?

Question

I tried to develop an FCN-16 model in Keras. I initialized the weights with similar FCN-16 model weights.

def FCN8 (nClasses, input_height=256, input_width=256):

    ## input_height and width must be devisible by 32 because maxpooling with filter size = (2,2) is operated 5 times,
    ## which makes the input_height and width 2^5 = 32 times smaller
    assert input_height % 32 == 0
    assert input_width % 32 == 0
    IMAGE_ORDERING = "channels_last"

    img_input = Input(shape=(input_height, input_width, 3))  ## Assume 224,224,3

    ## Block 1
    x = Conv2D(64, (3, 3), activation='relu', padding='same', name='conv1_1', data_format=IMAGE_ORDERING)(
        img_input)
    x = Conv2D(64, (3, 3), activation='relu', padding='same', name='conv1_2', data_format=IMAGE_ORDERING)(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool', data_format=IMAGE_ORDERING)(x)
    f1 = x

    # Block 2
    x = Conv2D(128, (3, 3), activation='relu', padding='same', name='conv2_1', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(128, (3, 3), activation='relu', padding='same', name='conv2_2', data_format=IMAGE_ORDERING)(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block2_pool', data_format=IMAGE_ORDERING)(x)
    f2 = x

    # Block 3
    x = Conv2D(256, (3, 3), activation='relu', padding='same', name='conv3_1', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(256, (3, 3), activation='relu', padding='same', name='conv3_2', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(256, (3, 3), activation='relu', padding='same', name='conv3_3', data_format=IMAGE_ORDERING)(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block3_pool', data_format=IMAGE_ORDERING)(x)
    pool3 = x

    # Block 4
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv4_1', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv4_2', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv4_3', data_format=IMAGE_ORDERING)(x)
    pool4 = MaxPooling2D((2, 2), strides=(2, 2), name='block4_pool', data_format=IMAGE_ORDERING)(
        x)  ## (None, 14, 14, 512)

    # Block 5
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv5_1', data_format=IMAGE_ORDERING)(pool4)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv5_2', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv5_3', data_format=IMAGE_ORDERING)(x)
    pool5 = MaxPooling2D((2, 2), strides=(2, 2), name='block5_pool', data_format=IMAGE_ORDERING)(
        x) 

    n = 4096
    o = (Conv2D(n, (7, 7), activation='relu', padding='same', name="fc6", data_format=IMAGE_ORDERING))(pool5)
    conv7 = (Conv2D(n, (1, 1), activation='relu', padding='same', name="fc7", data_format=IMAGE_ORDERING))(o)

    conv7 = (Conv2D(nClasses, (1, 1), activation='relu', padding='same', name="conv7_1", data_format=IMAGE_ORDERING))(conv7)

    conv7_4 = Conv2DTranspose(nClasses, kernel_size=(2, 2), strides=(2, 2),  data_format=IMAGE_ORDERING)(
        conv7)

    pool411 = (
        Conv2D(nClasses, (1, 1), activation='relu', padding='same', name="pool4_11",use_bias=False, data_format=IMAGE_ORDERING))(pool4)

    o = Add(name="add")([pool411, conv7_4])

    o = Conv2DTranspose(nClasses, kernel_size=(16, 16), strides=(16, 16), use_bias=False, data_format=IMAGE_ORDERING)(o)
    o = (Activation('softmax'))(o)

    GDI= Model(img_input, o)
    GDI.load_weights(Model_Weights_path)

    model = Model(img_input, o)

    return model

Then I did train, test split and trying to run the model as:

from keras import optimizers

sgd = optimizers.SGD(lr=1E-2, momentum=0.91,decay=5**(-4), nesterov=True)

model.compile(optimizer='sgd',loss='categorical_crossentropy',metrics=['accuracy'],)

hist1 = model.fit(X_train,y_train,validation_data=(X_test,y_test),batch_size=32,epochs=1000,verbose=2)

model.save("/content/drive/My Drive/HCI_prep/new.h5")

But this code is throwing error in the first epoch:

NotFoundError: 2 root error(s) found. (0) Not found: No algorithm worked! [[{{node pool4_11_3/Conv2D}}]] [[loss_4/mul/_629]] (1) Not found: No algorithm worked! [[{{node pool4_11_3/Conv2D}}]] 0 successful operations. 0 derived errors ignored.

enter image description here

See this: https://medium.com/@adwin596/solved-error-failed-to-get-convolution-algorithm-4396982082a7 — Geeocode, Dec 15 '19 at 11:43
@Niloy Chakraborty, Can you please confirm if the error is resolved by adding `padding=same` to maxpooling layers so that we can mention it as an answer for the benefit of the community. Else, can you please share complete traceback so that we can help you.Thanks! — , Jun 04 '20 at 11:28
@Tensorflow Warriors, have a same problem. Building custom UNet. Padding have no difference, for me. I checked link from Geeocode, to sum up i have cudadnn installed on my windows workstation, so it's no help. — Alexandr Crit, Jun 29 '20 at 06:50
Thank you, everyone. @TensorflowWarrior I have added the code in an answer section — Niloy Chakraborty, Sep 12 '20 at 14:29

score 12 · Answer 1 · edited Jan 19 '21 at 16:05

12

add the following to your code:

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession

config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

And then restart the python kernel.

edited Jan 19 '21 at 16:05

Community

1
1

answered Sep 21 '20 at 20:05

Oktay Alizada

290
1
6
19

2

Cool! Thanks, worked also for me. What is the explanation why this helps? – 2Obe May 25 '21 at 20:03

score 9 · Answer 2 · answered Sep 08 '20 at 08:34

9

Had the same issue.

The padding='same' for MaxPooling didn't work for me.

I changed the color_mode parameter in the train and test generators from 'rgb' to 'grayscale' and then it worked for me.

answered Sep 08 '20 at 08:34

Amit Sharma

690
8
22

score 8 · Answer 3 · answered May 06 '21 at 20:34

8

This worked for me:

    import tensorflow as tf
    physical_devices = tf.config.list_physical_devices('GPU')
    tf.config.experimental.set_memory_growth(physical_devices[0], True)

answered May 06 '21 at 20:34

Sohrab

181
2
4

This line should be now `physical_devices = tf.config.experimental.list_physical_devices('GPU')` – Sachin Mohan May 02 '22 at 08:47

score 5 · Answer 4 · answered Dec 02 '20 at 10:39

5

In my case, this was solved by ending all processes, that still allocated memory on one of the GPUs. Apparently, one of them did not finish (correctly). I did not have to change any code.

answered Dec 02 '20 at 10:39

Christian Steinmeyer

824
11
23

kill the process owning the gpu is a right choice – dtlam26 May 26 '22 at 08:32

score 5 · Answer 5 · answered Mar 21 '21 at 08:52

5

My problem was that I called the model with an input_shape of (?,28,28,1) and later called it with (?,28,28,3).

answered Mar 21 '21 at 08:52

Sak

109
2
2

score 1 · Answer 6 · edited May 29 '23 at 21:23

For reference, the full code that fixed the error is as follows:

import tensorflow.keras
from tensorflow.keras.models import *
from tensorflow.keras.layers import *

IMAGE_ORDERING = 'channels_last'

# take vgg-16 pretrained model from "https://github.com/fchollet/deep-learning-models" here
pretrained_url = "https://github.com/fchollet/deep-learning-models/" \
                 "releases/download/v0.1/" \
                 "vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5"

pretrained = 'imagenet'  # 'imagenet' if weights need to be initialized!

"""
Function Name: get_vgg_encoder()
Functionalities: This function defines the VGG encoder part of the FCN network
                 and initialize this encoder part with VGG pretrained weights.
Parameter:input_height=224,  input_width=224, pretrained=pretrained
Returns: final layer of every blocks as f1,f2,f3,f4,f5
"""


def get_vgg_encoder(input_height=224, input_width=224, pretrained=pretrained):
    pad = 1

    # heights and weights must be divided by 32, for fcn
    assert input_height % 32 == 0
    assert input_width % 32 == 0

    img_input = Input(shape=(input_height, input_width, 3))

    # Unlike base paper, stride=1 has not been used here, because
    # Keras has default stride=1

    x = (ZeroPadding2D((pad, pad), data_format=IMAGE_ORDERING))(img_input)
    x = Conv2D(64, (3, 3), activation='relu', padding='valid', name='block1_conv1', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2', data_format=IMAGE_ORDERING)(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool', data_format=IMAGE_ORDERING)(x)
    f1 = x
    # Block 2
    x = Conv2D(128, (3, 3), activation='relu', padding='same', name='block2_conv1', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(128, (3, 3), activation='relu', padding='same', name='block2_conv2', data_format=IMAGE_ORDERING)(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block2_pool', data_format=IMAGE_ORDERING)(x)
    f2 = x

    # Block 3
    x = Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv1', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv2', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv3', data_format=IMAGE_ORDERING)(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block3_pool', data_format=IMAGE_ORDERING)(x)
    x = Dropout(0.5)(x)
    f3 = x

    # Block 4
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv1', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv2', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv3', data_format=IMAGE_ORDERING)(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block4_pool', data_format=IMAGE_ORDERING)(x)
    f4 = x

    # Block 5
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv1', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv2', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv3', data_format=IMAGE_ORDERING)(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block5_pool', data_format=IMAGE_ORDERING)(x)
    # x= Dropout(0.5)(x)

    f5 = x

    # Check if weights are initialised, model is learning!
    if pretrained == 'imagenet':
        VGG_Weights_path = tensorflow.keras.utils.get_file(
            pretrained_url.split("/")[-1], pretrained_url)

        Model(img_input, x).load_weights(VGG_Weights_path)

    return img_input, [f1, f2, f3, f4, f5]


"""
Function Name: fcn_16()
Functionalities: This function defines the Fully Convolutional part of the FCN network
                 and adds skip connections to build FCN-16 network
Parameter:n_classes, encoder=get_vgg_encoder, input_height=224,input_width=224
Returns: model
"""


def fcn_16(n_classes, encoder=get_vgg_encoder, input_height=224, input_width=224):
    # Take levels from the base model, i.e. vgg
    img_input, levels = encoder(input_height=input_height, input_width=input_width)
    [f1, f2, f3, f4, f5] = levels

    o = f5

    # fcn6
    o = (Conv2D(4096, (7, 7), activation='relu', padding='same', data_format=IMAGE_ORDERING))(o)
    o = Dropout(0.5)(o)

    # fc7
    o = (Conv2D(4096, (1, 1), activation='relu', padding='same', data_format=IMAGE_ORDERING))(o)
    o = Dropout(0.3)(o)

    conv7 = (Conv2D(1, (1, 1), activation='relu', padding='same', name="score_sal", data_format=IMAGE_ORDERING))(o)

    conv7_4 = Conv2DTranspose(1, kernel_size=(4, 4), strides=(2, 2), padding='same', name="upscore_sal2",
                              use_bias=False, data_format=IMAGE_ORDERING)(conv7)

    pool411 = (
        Conv2D(1, (1, 1), activation='relu', padding='same', name="score_pool4", data_format=IMAGE_ORDERING))(f4)

    # Add a crop layer 
    o, o2 = crop(pool411, conv7_4, img_input)

    # add skip connection
    o = Add()([o, o2])

    # 16 x upsample
    o = Conv2DTranspose(n_classes, kernel_size=(32, 32), strides=(16, 16), use_bias=False, data_format=IMAGE_ORDERING)(
        o)

    # crop layer
    ## Caffe calls crop layer that takes o and img_input as argument, it takes their difference and crops
    ## But keras takes it as touple, I checked the size diff and put this value manually.
    ## output dim was 240 , input dim was 224. 240-224=16. so 16/2=8

    score = Cropping2D(cropping=((8, 8), (8, 8)), data_format=IMAGE_ORDERING)(o)

    o = (Activation('sigmoid'))(score)
    model = Model(img_input, o)

    model.model_name = "fcn_16"

    return model

score 0 · Answer 7 · answered Mar 31 '21 at 19:06

This error is quite general and basically indicates that "something" went wrong. As, the variety of answers suggest the error can arise from incompatibilities of the implementation with the underlying versions of keras/tensorflow, or the filter sizes are incorrect, or or or...

There is no single solution to this. For me, it also was an input shape issue. Instead of using rgb using grayscale worked as the network expected 1 channel.

score 0 · Answer 8 · answered May 29 '23 at 21:20

Like @Soren said, this error depends on various situations. It could happen due to VRAM deficiencies, v1, and v2 incompatibilities, shape issues while calling conv or pool, etc.

In my case, this error was arising because I was calling the saved model for inference in Python 3.6 (and TF 2.4.2), while my model was trained in Python 3.10 (and TF 2.8), and some flexible shape operations done in the functional API of Keras/TF was not backward compatible to an older version of TF.

Thus, I would also recommend checking your training and inference environment to make sure there are no bugs or mismatches.

How to solve "No Algorithm Worked" Keras Error?

8 Answers8

Linked