gRPC channel.close() taking 10 seconds

Question

I'm trying to use the following code in my production server(which I want to scale it to more than 500 TPS). I'm facing the following issue when I flood the server with many requests. In atleast 1 request in 1000 requests the channel.close() call is taking 10-10.5 seconds. I'm running the code on Flask server. Currently, for every request I'm creating a channel and closing it. Please help me with this.

channel = grpc.insecure_channel(serving_address)
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)
request = predict_pb2.PredictRequest()
request.model_spec.name = model_name
request.model_spec.signature_name = 'serving_default'
request.inputs['model_2_input'].CopyFrom(
        make_tensor_proto_engine(img_array, dtype=1, shape=[1, 224, 224, 3]))
result = stub.Predict(request, 6.0)
channel.close()

Hello Hareesh, Couple of points. 1. Can you try using Tensorflow Model Server and check if the issue still persists. 2. Do you have any specific reason for closing the Channel because, that is not being done in the code in the link, https://github.com/tensorflow/serving/blob/master/tensorflow_serving/example/mnist_client.py. — , Nov 04 '19 at 08:54
Hi @TensorflowSupport, 1. Yes we are using a Tf-Serving container on our production server. 2. I think there is a default timeout set for a channel on server side(ie: tf serving container). I used channel.close() to overcome that issue. — Sathu Hareesh Babu, Nov 07 '19 at 20:51
@TensorflowSupport also could you please tell me how to add request logs on Tensorflow Model Server(tensorflow_serving container) — Sathu Hareesh Babu, Nov 07 '19 at 20:54
You can use "--v=1" for the Logs. Refer https://github.com/tensorflow/serving/issues/285 and https://stackoverflow.com/questions/46359852/logging-requests-being-served-by-tensorflow-serving-model for more info. Yes, Logs may help us understand the issue. — , Nov 08 '19 at 08:24

gRPC channel.close() taking 10 seconds

0 Answers0