Invalid character found in base64 while using a deployed model on cloudml

Question

For better context, I have uploaded a pre-trained model on cloud ml. It's an inceptionV3 model converted from keras to acceptable format in tensorflow.

from keras.applications.inception_v3 import InceptionV3
model = InceptionV3(weights='imagenet') 
from keras.models import Model
intermediate_layer_model = Model(inputs=model.input,outputs=model.layers[311].output) 
with tf.Graph().as_default() as g_input:
    input_b64 = tf.placeholder(shape=(1,),
                               dtype=tf.string,
                               name='input')
    input_bytes = tf.decode_base64(input_b64[0])
    image = tf.image.decode_image(input_bytes)
    image_f = tf.image.convert_image_dtype(image, dtype=tf.float32)
    input_image = tf.expand_dims(image_f, 0)
    output = tf.identity(input_image, name='input_image') 
g_input_def = g_input.as_graph_def()
K.set_learning_phase(0)
sess = K.get_session()
from tensorflow.python.framework import graph_util
g_trans = sess.graph
g_trans_def = graph_util.convert_variables_to_constants(sess,
    g_trans.as_graph_def(),
    [intermediate_layer_model.output.name.replace(':0','')])
with tf.Graph().as_default() as g_combined:
    x = tf.placeholder(tf.string, name="input_b64")

    im, = tf.import_graph_def(g_input_def,
        input_map={'input:0': x},
        return_elements=["input_image:0"])

    pred, = tf.import_graph_def(g_trans_def,
             input_map={intermediate_layer_model.input.name: im,
             'batch_normalization_1/keras_learning_phase:0': False},
             return_elements=[intermediate_layer_model.output.name])

    with tf.Session() as sess2:
        inputs = {"inputs": tf.saved_model.utils.build_tensor_info(x)}
        outputs = {"outputs":tf.saved_model.utils.build_tensor_info(pred)}
        signature =tf.saved_model.signature_def_utils.build_signature_def(
                inputs=inputs,
                outputs=outputs,
        method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME
            )

      # save as SavedModel
        b = tf.saved_model.builder.SavedModelBuilder('inceptionv4/')
        b.add_meta_graph_and_variables(sess2,
                      [tf.saved_model.tag_constants.SERVING],
                      signature_def_map={'serving_default': signature})
        b.save()

The generated pb file works fine when I use it locally. But when I deploy it on cloud ml I get the following error.

RuntimeError: Prediction failed: Error during model execution: AbortionError(code=StatusCode.INVALID_ARGUMENT, details="Invalid character found in base64.
     [[Node: import/DecodeBase64 = DecodeBase64[_output_shapes=[<unknown>], _device="/job:localhost/replica:0/task:0/device:CPU:0"](import/strided_slice)]]")

Following is the code I use for getting local predictions.

import base64
import json

with open('MEL_BE_0.jpg', 'rb') as image_file:
    encoded_string = str(base64.urlsafe_b64encode(image_file.read()),'ascii')

import tensorflow as tf

with tf.Session(graph=tf.Graph()) as sess:
    MetaGraphDef=tf.saved_model.loader.load(
       sess,
       [tf.saved_model.tag_constants.SERVING],
       'inceptionv4')
    input_tensor = tf.get_default_graph().get_tensor_by_name('input_b64:0')
    print(input_tensor)
    avg_tensor = tf.get_default_graph().get_tensor_by_name('import_1/avg_pool/Mean:0')
    print(avg_tensor)
    predictions = sess.run(avg_tensor, {input_tensor: [encoded_string]})

And finally following is the code snippet that I use for wrapping the encoded string in the request that is sent to the cloud-ml engine.

request_body= json.dumps({"key":"0", "image_bytes": {"b64": [encoded_string]}})

Answer 1

It looks like you are trying to do the base64 decoding in TensorFlow and use the {"b64": ...} JSON format. You need to do one or the other; we typically recommend the latter.

As a side note, your input placeholder must have an outer dimension of None. That can make some things tricky, e.g., you'll either have to reshape the dimensions to be size 1 (which will prevent you from using the batch prediction service in its current state) or you'll have to us tf.map_fn to apply the same set of transformations to each element of the input "batch". You can find an example of that technique in this example.

Finally, I recommend the use of tf.saved_model.simple_save.

Putting it altogether, here is some modified code. Note that I'm inlining your input function (as opposed to serializing it to a graph def and reimporting):

HEIGHT = 299
WIDTH = 299

# Get Keras Model
from keras.applications.inception_v3 import InceptionV3
model = InceptionV3(weights='imagenet') 
from keras.models import Model
intermediate_layer_model = Model(inputs=model.input,outputs=model.layers[311].output) 
K.set_learning_phase(0)
sess = K.get_session()
from tensorflow.python.framework import graph_util
g_trans = sess.graph
g_trans_def = graph_util.convert_variables_to_constants(sess,
    g_trans.as_graph_def(),
    [intermediate_layer_model.output.name.replace(':0','')])

# Create inputs to model and export
with tf.Graph().as_default() as g_combined:

  def decode_and_resize(image_bytes):
    image = tf.image.decode_image(image_bytes)
    # Note resize expects a batch_size, but tf_map supresses that index,
    # thus we have to expand then squeeze.  Resize returns float32 in the
    # range [0, uint8_max]
    image = tf.expand_dims(image, 0)
    image = tf.image.resize_bilinear(
        image, [HEIGHT, WIDTH], align_corners=False)
    image = tf.squeeze(image, squeeze_dims=[0])
    image = tf.cast(image, dtype=tf.uint8)
    return image

  input_byes = tf.placeholder(shape=(None,),
                             dtype=tf.string,
                             name='input')

  images = tf.map_fn(
      decode_and_resize, input_bytes, back_prop=False, dtype=tf.uint8)
  images = tf.image.convert_image_dtype(images, dtype=tf.float32)

  pred, = tf.import_graph_def(g_trans_def,
         input_map={intermediate_layer_model.input.name: images,
         'batch_normalization_1/keras_learning_phase:0': False},
         return_elements=[intermediate_layer_model.output.name])

  with tf.Session() as sess2:
      tf.saved_model.simple_save(
          sess2,
          model_dir='inceptionv4/'
          inputs={"inputs": input_bytes},
          outputs={"outputs": pred})

Note: I'm not 100% certain that the shapes of intermediate_layer_model and images are compatible. The shape of images will be [None, height, width, num_channels].

Also note that your local prediction code will change a bit. You don't base64 encode the images and you need to send a "batch"/list of images rather than single images. Something like:

with open('MEL_BE_0.jpg', 'rb') as image_file:
  encoded_string = image_file.read()

input_tensor = tf.get_default_graph().get_tensor_by_name('input:0')
print(input_tensor)
avg_tensor = tf.get_default_graph().get_tensor_by_name('import_1/avg_pool/Mean:0')
print(avg_tensor)
predictions = sess.run(avg_tensor, {input_tensor: [encoded_string]})

You didn't specify whether you're doing batch prediction or online prediction, which have similar but slightly different "formats" for the inputs. In either case, your model is not exporting a "key" field (did you mean to? It's probably helpful for batch prediction, but not for online).

For batch prediction, the file format is JSON lines; each line contains one example. Each line can be generated like so from Python:

example = json.dumps({"image_bytes": {"b64": ENCODED_STRING}})

(Note the omission of "key" for now). Since you only have one input, there is a shorthand:

example = json.dumps({"b64": ENCODED_STRING})

If you want to do online prediction, you'll note that if you are using gcloud to send requests, you actually use the same file format as for batch prediction.

In fact, we highly recommend using gcloud ml-engine local predict --json-instances=FILE --model-dir=... before deploying to the cloud to help debug.

If you intend to use some other client besides gcloud, e.g., in a web app, mobile app, frontend server, etc., then you won't be sending a file and you need to construct the full request yourself. It's very similar to the file format above. Basically, take each line of the JSON lines file and put them in an array calle "instances", i.e.,

request_body= json.dumps({"instances": [{"image_bytes": {"b64": [encoded_string]}}]})

You can use the same syntactic sugar if you'd like:

request_body= json.dumps({"instances": [{"b64": [encoded_string]}]})

I hope this helps!