How to access content of repeated message data

Question

I want to use protocol buffer to send and receive the following type using gRPC

std:array<std::complex<int>, 2> bar_array;

• Sources used to get idea: 1, 2

What I have done so far

My approach (Intentionally I am omitting here the unnecessary code)

• proto file

syntax = "proto3";

package expcmake;

message child_data {
    repeated int32 child_data_list = 1;
}

message Address {
    child_data child_data_var = 8;
    repeated child_data parent_data_list = 9;
}

• server.cpp

  • Here, at first I have made a dummy std:array<std::complex<int>, 2> data. Then, I have filled the child_data_list with the std::complex<int> data. After each filling of real and imaginary part I have pushed them in the parent_data_list. Also, at this moment I have cleared the child_data_list.
  • Client message name is NameQuery, while Server message name is Address
  • In Server side both message are passed as pointer

class AddressBookService final : public expcmake::AddressBook::Service {
    public:
        virtual ::grpc::Status GetAddress(::grpc::ServerContext* context, const ::expcmake::NameQuery* request, ::expcmake::Address* response)
{

// omitting unnecessary lines

// populate bar_array with std::complex<int> data
std::complex<int> z4 = 1. + 2i, z5 = 1. - 2i; // conjugates
bar_array = {z4, z5};

std::cout << "bar array printing whose size: " << bar_array.size() << std::endl;
for(int i = 0; i < bar_array.size(); i++) {
    std::cout << bar_array[i] << " ";
}
std::cout << std::endl;

// Use parent_data_list protocol buffer message type to fill with the content of bar_array
for (int i = 0; i < bar_array.size(); i++){
    
    // Use child_data protocol buffer message type to fill with the content of complex int data
    response->mutable_child_data_var()->add_child_data_list(real(bar_array[i]));
    response->mutable_child_data_var()->add_child_data_list(imag(bar_array[i]));
    
    // Use parent_data_list protocol buffer message type to fill with the content of child_data -> child_data_list data
    response->add_parent_data_list() -> child_data_list();
    
    // clear the child_data message. Reason to keep child_data_list new in every iteration otherwise add_child_data_list will append new data (eg: 1,2,1,-2) which is wrong. Expected is (1,2) then (1,-2)
    response->mutable_child_data_var()->clear_child_data_list();
}

// This is zero which I have got. Without clearing it is 4 which is also correct I believe as per the concept of protocol buffer message type
std::cout << "response->mutable_child_data_var()->child_data_list_size(): " << response->mutable_child_data_var()->child_data_list_size() << std::endl; 

// This is 2 which meets my requirement
std::cout << "response->parent_data_list_size(): " << response->parent_data_list_size() << std::endl;

// omitting unnecessary lines

}
};

• client.cpp

int main(int argc, char* argv[])
{
    // Setup request
    expcmake::NameQuery query;
    expcmake::Address result;
    // printing the content of child_data -> child_data_list data array/container. There I have seen 1,2,1,-2 if I don't do the clear operation on child_data_list in server side. So, I guess it is correctly got the data 
          for(int i = 0; i < result.mutable_child_data_var()->child_data_list_size(); i++)
            std::cout << "Child Data at index [" << i << "]: " << result.mutable_child_data_var()->child_data_list(i) << std::endl;

    // This one making problem
    // printing the content of parent_data_list type/container
    // for(int i = 0; i < result.parent_data_list_size(); i++){
    //   std::cout << "Parent Data at index [" << i << "]: " << result.mutable_parent_data_list(i) << std::endl; // This give me the memory address

    // Tried others to fetch the data but failed. Eg: result.parent_data_list(i) // failed
    // 
    // }
}

• Snippet from the generated pb file

  // repeated int32 child_data_list = 1;
  int child_data_list_size() const;
  private:
  int _internal_child_data_list_size() const;
  public:
  void clear_child_data_list();
  private:
  ::PROTOBUF_NAMESPACE_ID::int32 _internal_child_data_list(int index) const;
  const ::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int32 >&
      _internal_child_data_list() const;
  void _internal_add_child_data_list(::PROTOBUF_NAMESPACE_ID::int32 value);
  ::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int32 >*
      _internal_mutable_child_data_list();
  public:
  ::PROTOBUF_NAMESPACE_ID::int32 child_data_list(int index) const;
  void set_child_data_list(int index, ::PROTOBUF_NAMESPACE_ID::int32 value);
  void add_child_data_list(::PROTOBUF_NAMESPACE_ID::int32 value);
  const ::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int32 >&
      child_data_list() const;
  ::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int32 >*
      mutable_child_data_list();


  // .expcmake.child_data child_data_var = 8;
  bool has_child_data_var() const;
  private:
  bool _internal_has_child_data_var() const;
  public:
  void clear_child_data_var();
  const ::expcmake::child_data& child_data_var() const;
  PROTOBUF_MUST_USE_RESULT ::expcmake::child_data* release_child_data_var();
  ::expcmake::child_data* mutable_child_data_var();
  void set_allocated_child_data_var(::expcmake::child_data* child_data_var);
  private:
  const ::expcmake::child_data& _internal_child_data_var() const;
  ::expcmake::child_data* _internal_mutable_child_data_var();
  public:
  void unsafe_arena_set_allocated_child_data_var(
      ::expcmake::child_data* child_data_var);
  ::expcmake::child_data* unsafe_arena_release_child_data_var();


  // repeated .expcmake.child_data parent_data_list = 9;
  int parent_data_list_size() const;
  private:
  int _internal_parent_data_list_size() const;
  public:
  void clear_parent_data_list();
  ::expcmake::child_data* mutable_parent_data_list(int index);
  ::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::expcmake::child_data >*
      mutable_parent_data_list();
  private:
  const ::expcmake::child_data& _internal_parent_data_list(int index) const;
  ::expcmake::child_data* _internal_add_parent_data_list();
  public:
  const ::expcmake::child_data& parent_data_list(int index) const;
  ::expcmake::child_data* add_parent_data_list();
  const ::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::expcmake::child_data >&
      parent_data_list() const;

I guess

• Is the filling of the message field is incorrectly done !! Though the size is not saying that
• I am not catching the protobuf syntax(which is generated in the pb file) in right way to fetch the data

Need suggestions(helpful if can provide the syntax too).

Answer 1

The mutable_* prefixed APIs are for mutating (modifying/adding) elements and they return a pointer. The mutable_* and set_* APIs should only be used while filling the data. Once data is filled, you should not be using mutable_* APIs to check sizes. After receiving the response from the server, the client will most probably be consuming it, not mutating. You need to update the client-side accordingly.

Also, you can use a range-based for loop to iterate over child_data_list of child_data_var like this:

const auto child_data_list = result.child_data_var().child_data_list();
for (const auto element : child_data_list)
{
    // process element
}

---

Apart from that, you're using std::array (a fixed-size array) i.e. std:array<std::complex<int>, 2>, alternatively here's another way to model this without repeated which does not represent a fixed-size array.

complex.proto

syntax = "proto3";

package pb;

message Complex {
    int32 real = 1;
    int32 imag = 2;
}

message Compound {
    Complex c1 = 1;
    Complex c2 = 2;
}

The message Complex emulates an std::complex type and Compound an std::array with only 2 elements c1 and c2.

With this, you can easily convert the protobuf message to std::complex and vice versa. Once converted to std::complex, you can perform its supported operations as needed.

• Compile complex.proto:

protoc --cpp_out=. ./complex.proto

For the C++ API, look for the accessors in the generated complex.pb.h file under public. The official doc Protocol Buffer Basics: C++ also lists the C++ API examples under The Protocol Buffer API section.

Here's a complete C++ example with the above complex.proto:

main.cpp

#include <iostream>
#include <complex>
#include <array>
#include <cstdlib>

#include "complex.pb.h"

namespace my {
    using complex = std::complex<int>;
    using compound = std::array<std::complex<int>, 2>;
}

int main()
{
    const auto my_c1 = my::complex{ 1, 2 };
    const auto my_c2 = my::complex{ 2, 4 };
    const auto my_compound_1 = my::compound{ my_c1, my_c2 };

    std::cout << "my_compound_1 [size: " << my_compound_1.size() << "]\n";
    std::cout << "my_c1: " << my_c1 << '\n';
    std::cout << "my_c2: " << my_c2 << '\n';

    pb::Compound pb_compound;
    pb_compound.mutable_c1()->set_real(my_compound_1[0].real());
    pb_compound.mutable_c1()->set_imag(my_compound_1[0].imag());
    pb_compound.mutable_c2()->set_real(my_compound_1[1].real());
    pb_compound.mutable_c2()->set_imag(my_compound_1[1].imag());

    std::cout << "\npb_compound:\n";
    pb_compound.PrintDebugString();

    const auto serialized_compound = pb_compound.SerializeAsString();

    // send

    // receive

    pb::Compound deserialized_compound;
    if (!deserialized_compound.ParseFromString(serialized_compound))
    {
        std::cerr << "[ERROR] Parsing failed!\n";
        return EXIT_FAILURE;
    }
    
    std::cout << "\n\npb_compound (deserialized):\n";
    deserialized_compound.PrintDebugString();

    const auto pb_c1 = deserialized_compound.c1();
    const auto pb_c2 = deserialized_compound.c2();

    const auto my_c3 = my::complex{ pb_c1.real(), pb_c1.imag() };
    const auto my_c4 = my::complex{ pb_c2.real(), pb_c2.imag() };
    const auto my_compound_2 = my::compound{ my_c3, my_c4 };

    std::cout << "my_compound_2 [size: " << my_compound_2.size() << "]\n";
    std::cout << "my_c3: " << my_c3 << '\n';
    std::cout << "my_c4: " << my_c4 << '\n';

    const auto sum = my_c3 + my_c4;
    std::cout << "sum: " << sum << '\n';

    return EXIT_SUCCESS;
}

• Compile with g++:

g++ main.cpp complex.pb.cc -o pb_complex `pkg-config --cflags --libs protobuf`

• Run:

$ ./pb_complex
my_compound_1 [size: 2]
my_c1: (1,2)
my_c2: (2,4)

pb_compound:
c1 {
  real: 1
  imag: 2
}
c2 {
  real: 2
  imag: 4
}


pb_compound (deserialized):
c1 {
  real: 1
  imag: 2
}
c2 {
  real: 2
  imag: 4
}
my_compound_2 [size: 2]
my_c3: (1,2)
my_c4: (2,4)
sum: (3,6)

Answer 2

The way I was accessing the filed parent_data_list was wrong. I am posting the approach which has solved the issue.

• A little change in proto file

syntax = "proto3";

package expcmake;

message child_data {
    repeated int32 child_data_list = 1 [packed=true];
}

message Address {
    repeated child_data parent_data_list = 1;
}

Reason to use packed see this

• server.cpp

class AddressBookService final : public expcmake::AddressBook::Service {
    public:
        virtual ::grpc::Status GetAddress(::grpc::ServerContext* context, const ::expcmake::NameQuery* request, ::expcmake::Address* response){

// omitting unnecessary lines

// populate bar_array with std::complex<int> data
std::complex<int> z4 = 1. + 2i, z5 = 1. - 2i; // conjugates
bar_array = {z4, z5};

// Now goal is to pass this bar_array as a protobuf message

// Use parent_data_list protocol buffer message type to fill with the content of bar_array
for (int i = 0; i < bar_array.size(); i++){

    // make a 2D array. Eg: std::array<std::complex<int>>. In each iteration, new sub_content will be added (here sub_content means child_data_list)
    response->add_parent_data_list() -> child_data_list();

    // Followings are filling the child_data_list with the required data(Real and Imag part of the complex data which is already generated)
    response->mutable_parent_data_list(i)->add_child_data_list(real(bar_array[i]));
    response->mutable_parent_data_list(i)->add_child_data_list(imag(bar_array[i]));
        }
    }
    return grpc::Status::OK;
};

After the completion of the iteration, A 2D type of message will be generated which is parent_data_list = [[child_data_list], [child_data_list]]. Don't consider it as a real scenario. Just for presentation I am using it.

• client.cpp

std::cout <<"Using Mutable" << std::endl;
for(int i = 0; i < result.parent_data_list_size(); i++){
    for(int j = 0 ; j<result.mutable_parent_data_list(i)->child_data_list_size(); j++){
        std::cout << "Value [" << i << "][" << j << "]: " << result.mutable_parent_data_list(i)->child_data_list(j) << std::endl;
    }
}

// Following is same as before. Added as I was in turmoil to understand the properties generated by protobuf. Hope could be helpful for beginners like me 
std::cout <<"Without Mutable property" << std::endl;
for(int i = 0; i < result.parent_data_list_size(); i++){
    for(int j = 0 ; j < result.parent_data_list(i).child_data_list_size(); j++){
        std::cout << "Value [" << i << "][" << j << "]: " << result.parent_data_list(i).child_data_list(j) << std::endl;
    }
}

In client side I am accessing the child_data_list with index which is not properly mimic the C++ style. Would be great if I can in one line print the value of parent_data_list[n] data. As far I have studied the generated pb.h file my understanding is that it is not possible as no return function I have not found without index for the repeated field. Or may be I am missing something important again.