FGSM attack/Keras gradient when the classification model is a SVM

Question

I am working on an FGSM attack for a Facenet classifier that uses a Keras model for the embedding and an SVM to come up with the classification. I am having a hard time determining how to format my inputs coming in since I need to pass the input first to the SVM before I can take the loss but then I am getting the gradient as None which fails as an error. Do you have any ideas on how I could pass the values effectively?

for i in range(epochs): 
    print(i)
    # One hot encode the target class
    target = K.one_hot(target_class, 5)

    # Get the new image and predictions
    embeddingPred=asarray(embed_model.output[0])
    prediction= tf.reshape(model.predict_proba(embeddingPred),[5,])
    prediction = tf.cast(tf.convert_to_tensor(prediction), tf.float32)

    # Get the loss and gradient of the loss wrt the inputs
    loss = -1*K.categorical_crossentropy(target, prediction)
    grads = K.gradients(loss, embed_model.input)

    # Get the sign of the gradient
    delta = K.sign(grads[0])
    x_noise = x_noise + delta

    # Perturb the image
    x_adv = x_adv + epsilon*delta

    # Get the new image and predictions
    x_adv = sess.run(x_adv, feed_dict={embed_model.input:x})