3

I have a Keras graph with a float32 tensor of shape (?, 224, 224, 3) that I want to export to Tensorflow Serving, in order to make predictions with RESTful. Problem is that I cannot input tensors, but encoded b64 strings, as that is a limitation of the REST API. That means that when exporting the graph, the input needs to be a string that needs to be decoded.

How can I "inject" the new input to be converted to the old tensor, without retraining the graph itself? I have tried several examples [1][2].

I currently have the following code for exporting: 

image = tf.placeholder(dtype=tf.string, shape=[None], name='source')


signature = predict_signature_def(inputs={'image_bytes': image},
                                 outputs={'output': model.output})

I somehow need to find a way to convert image to model.input, or a way to get the model output to connect to image.

Any help would be greatly appreciated!

user3337758
  • 115
  • 1
  • 12

2 Answers2

2

You can use tf.decode_base64:

image = tf.placeholder(dtype=tf.string, shape=[None], name='source')
image_b64decoded = tf.decode_base64(image)
signature = predict_signature_def(inputs={'image_bytes': image_b64decoded},
                                 outputs={'output': model.output})

EDIT:

If you need to use tf.image.decode_image, you can get it to work with multiple inputs using tf.map_fn:

image = tf.placeholder(dtype=tf.string, shape=[None], name='source')
image_b64decoded = tf.decode_base64(image)
image_decoded = tf.map_fn(tf.image.decode_image, image_b64decoded, dtype=tf.uint8)

This will work as long as the images have all the same dimensions, of course. However, the result is a tensor with completely unknown shape, because tf.image.decode_image can output a different number of dimensions depending on the type of image. You can either reshape it or use another tf.image.decode_* call so at least you have a known number of dimensions in the tensor.

jdehesa
  • 58,456
  • 7
  • 77
  • 121
  • I'm not sure how is exactly your model, so I assumed the decoded base64 data is the input that you need. If you need image decoding you can also use [`tf.image.decode_image`](https://www.tensorflow.org/api_docs/python/tf/image/decode_image) or related, or if you need reshaping just [`tf.reshape`](https://www.tensorflow.org/api_docs/python/tf/reshape). – jdehesa Aug 07 '18 at 14:05
  • tf.image.decode_image would be what I am looking for, but unfortunately, it can only operate on shapes of rank 0... – user3337758 Aug 07 '18 at 14:39
  • The problem is that I need to have the input in the form of a b64 string, but the graph architecture is trained to have a float tensor input shape. That means that the inputs need to be in the form of a string as well... – user3337758 Aug 07 '18 at 15:14
  • @user3337758 I'm sorry but I don't understand well the issue. You can, within TensorFlow, take a list of Base64 strings, decode them, decode the byte arrays into images, reshaped them if you need and change the image data type if necessary (e.g. with [`tf.image.convert_image_dtype`](https://www.tensorflow.org/api_docs/python/tf/image/convert_image_dtype). I'm not sure what is the missing step. – jdehesa Aug 07 '18 at 15:21
  • The missing step would be connecting this result to the initial float32 tensor of the model. How would I go about doing that? – user3337758 Oct 05 '18 at 14:07
  • @user3337758 So what you mean is you already have a graph in place, and now you want to sort of "prepend" this preprocessing to it? Unfortunately that is not easy, it would have been simpler to have the additional first steps since the beginning and feed values at some point in the middle for training (you can feed values at any point in the graph, not just placeholders). If you want to edit the connections in a graph, you could use the [Graph Editor module](https://www.tensorflow.org/api_guides/python/contrib.graph_editor), but it is not very straightforward. – jdehesa Oct 05 '18 at 15:49
  • Thanks for the clarification. I will retrain the modified graph then. – user3337758 Oct 11 '18 at 08:32
  • @user3337758 Thinking further about this, I think you may also be able to achieve what you want with [`tf.import_graph_def`](https://www.tensorflow.org/api_docs/python/tf/import_graph_def). This would allow you to export a trained model as a graph definition, with [`tf.Graph.as_graph_def()`](https://www.tensorflow.org/api_docs/python/tf/Graph#as_graph_def), and the reimport it in another graph, connecting the inputs to other tensors with the `input_map` parameter. Although, to do this, the original graph would have to be frozen if you want to save the trained weights. – jdehesa Oct 12 '18 at 15:07
-1

Creating an export_model may be an easier way. One example in tensorflow.org

  1. The Keras graph with a float32, shape (?, 224, 224, 3) tensor

model = ...

  1. Define a function to preprocess b64 image
def preprocess_input(base64_input_bytes):
    def decode_bytes(img_bytes):
        img = tf.image.decode_jpeg(img_bytes, channels=3)
        img = tf.image.resize(img, (224, 224))
        img = tf.image.convert_image_dtype(img, tf.float32)
        return img

    base64_input_bytes = tf.reshape(base64_input_bytes, (-1,))
    return tf.map_fn(lambda img_bytes:
                     decode_bytes(img_bytes),
                     elems=base64_input_bytes,                     
                     fn_output_signature=tf.float32)
  1. Export a serving model
serving_inputs = tf.keras.layers.Input(shape=(), dtype=tf.string, name='b64_input_bytes')
serving_x = tf.keras.layers.Lambda(preprocess_input, name='decode_image_bytes')(serving_inputs)
serving_x = model(serving_x)
serving_model = tf.keras.Model(serving_inputs, serving_x)
tf.saved_model.save(serving_model, serving_model_path)
  1. Serving
import requests
data = json.dumps({"signature_name": "serving_default", "instances": [{"b64_input_bytes": {"b64": b64str_1}}, {"b64_input_bytes": {"b64": b64str_2}}]})
headers = {"content-type": "application/json"}
json_response = requests.post('http://localhost:8501/v1/models/{model_name}:predict', data=data, headers=headers)
predictions = json.loads(json_response.text)['predictions']
reedcolab
  • 1
  • 1
  • Since you have a Lambda layer, you can't invoke the exported model without mentioning the custom function; `preprocess_input`. if you have any other options of how to save the model with the custom function which passed to the Lambda Layer, Please provide it here. – Ahmed Jan 02 '22 at 21:49
  • @Dr.Xavier Thanks for comment. The custom function (preprocess_input) is mentioned in the serving_x, and we just create "a new model" which includes the custom function and trained model. (e.g., Trained Model -> Export Model(Processing, Trained Model) ) – reedcolab Jan 07 '22 at 01:57