1

I have successfully trained a Keras model and used it for predictions on my local machine, now i want to deploy it using Tensorflow Serving. My model takes images as input and returns a mask prediction.

According to the documentation here my instances need to be formatted like this: 

{'image_bytes': {'b64': base64.b64encode(jpeg_data).decode()}}

Now, the saved_model.pb file automatically saved by my Keras model has the following tensor names:

input_tensor = graph.get_tensor_by_name('input_image:0')
output_tensor = graph.get_tensor_by_name('conv2d_23/Sigmoid:0')

therefore i need to save a new saved_model.pb file with a different signature_def.

I tried the following (see here for reference), which works:

with tf.Session(graph=tf.Graph()) as sess:
    tf.saved_model.loader.load(sess, ['serve'], 'path/to/saved/model/')
    graph = tf.get_default_graph()
    input_tensor = graph.get_tensor_by_name('input_image:0')
    output_tensor = graph.get_tensor_by_name('conv2d_23/Sigmoid:0')
    tensor_info_input = tf.saved_model.utils.build_tensor_info(input_tensor) 
    tensor_info_output = tf.saved_model.utils.build_tensor_info(output_tensor)
    prediction_signature = (
        tf.saved_model.signature_def_utils.build_signature_def(
            inputs={'image_bytes': tensor_info_input},
            outputs={'output_bytes': tensor_info_output},
method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME))
    builder = tf.saved_model.builder.SavedModelBuilder('path/to/saved/new_model/')
    builder.add_meta_graph_and_variables(
        sess, [tf.saved_model.tag_constants.SERVING],
        signature_def_map={'predict_images': prediction_signature, })
    builder.save()

but when i deploy the model and request predictions to the AI platform, i get the following error:

RuntimeError: Prediction failed: Error processing input: Expected float32, got {'b64': 'Prm4OD7JyEg+paQkPrGwMD7BwEA'} of type 'dict' instead.

readapting the answer here, i also tried to rewrite

input_tensor = graph.get_tensor_by_name('input_image:0')

as

image_placeholder = tf.placeholder(tf.string, name='b64')
graph_input_def = graph.as_graph_def()
input_tensor, = tf.import_graph_def(
    graph_input_def,
    input_map={'b64:0': image_placeholder},
    return_elements=['input_image:0'])

with the (wrong) understanding that this would add a layer on top of my input tensor with matching 'b64' name (as per documentation) that accepts a string and connects it the original input tensor but the error from the AI platform is the same.

(the relevant code i use for requesting a prediction is:

instances = [{'image_bytes': {'b64': base64.b64encode(image).decode()}}]
response = service.projects().predict(
    name=name,
    body={'instances': instances}
).execute()

where image is a numpy.ndarray of dtype('float32'))

I feel i'm close enough but i'm definitely missing something. Can you please help?

ansi
  • 59
  • 1
  • 8

1 Answers1

0

After b64 encoded -> decoded, the buffer of img will be changed to type string and not fit your model input type. You may try to add some preprocess in your model and send b64 request again.

zzachimonde
  • 455
  • 3
  • 8
  • Thank you for your answer. I made some progress but still struggling, i created a new question for my current specific problem: https://stackoverflow.com/questions/56275522/tensorflow-serving-how-to-decode-base64-encoded-png-images-during-preprocessing – ansi May 23 '19 at 13:47