5

I want to deploy a big model, e.g. bert, on spark to do inference since I don't have enough GPUs. Now I have two problems.

  1. I export the model to be pb format and load the model using the SavedModelBundle interface.
SavedModelBundle bundle=SavedModelBundle.load("E:\\pb\\1561992264","serve");

However, I can't find a way to load a pb model for hdfs filesystem path

  1. The spark environment's Glibc version isn't compatible with the tensorflow version I trained the model. Anyway to go around this?

I am not sure this is a good way to serving a tensorflow model on spark. Any other suggestions are appreciated!

Innat
  • 16,113
  • 6
  • 53
  • 101
K_Augus
  • 372
  • 2
  • 14

1 Answers1

-1

You could use Elephas (https://github.com/maxpumperla/elephas), which enables distributed training and inference of Keras models on Spark. Since you mentioned it's a Tensorflow model, this may require a conversion (detailed here: How can I convert a trained Tensorflow model to Keras?), but once it is a Keras model, it should be as simple as:

from elephas.spark_model import SparkModel


model = ... # load Keras model
data = ... # load in the data you want to perform inference on
spark_model = SparkModel(model)
predictions = spark_model.predict(data) # perform distributed inference on Spark cluster or local cores, depending on how Spark session is configured
danielcahall
  • 2,672
  • 8
  • 14