0

Is there a good pattern for sending a large dataset using gRPC and protocol buffers that has a mix of some header data, and some large repeated data?

E.g. for a server that accepts a matrix generated by some other process as input, a service and message might look as follows:

service MatrixService {
    rpc DoSomething(stream Matrix) returns (stream Matrix) {}
}

message Matrix {
    uint32 num_rows = 1;
    uint32 num_cols = 2;
    string created_by = 3;
    string creation_parameters = 4;
    repeated float data = 5;
}

It's really only the data field that is large enough to require streaming, the rest of the parameters are just headers that the server only needs to receive once.

Is there some commonly used pattern for efficiently making gRPC request that contains some initial header information, and a large amount of repeated data?

Should oneOf be used in cases like this (i.e. splitting the Matrix message into a oneOf { MatrixHeader, MatrixData }?

Or is it typically more common to just set the header fields on the first request, and leave them blank by convention in subsequent requests?

Or are there some other solutions that I've not considered?

Dave Challis
  • 3,525
  • 2
  • 37
  • 65

1 Answers1

1

You might want to consider chunking your data. See gRPC + Image Upload. Also, note that the maximum receive message size is 4 MB by default. You can increase it by setting the channel argument GRPC_ARG_MAX_RECEIVE_MESSAGE_LENGTH

user3126412
  • 326
  • 1
  • 4