Copy a std::vector to a repeated field from protobuf with memcpy

Question

At first I have this simple protobuf file

message messagetest
{
    ...
    repeated float samples = 6;
    ....
}

Which creates a headerfile with this methods

    //repeated float samples = 6;
      inline int samples_size() const;
      inline void clear_samples();
      static const int kSamplesFieldNumber = 6;
      inline float samples(int index) const;
      inline void set_samples(int index, float value);
      inline void add_samples(float value);
      inline const ::google::protobuf::RepeatedField< float >&  samples() const;
      inline ::google::protobuf::RepeatedField< float >* mutable_samples();

What I'm basically doing is to copy all data one by one in a for loop.

int main(int argc, char** argv)
{    
    messagetest fMessage;
    
    vector<float> fData (1000, 0);

    // Create 1000 random values
    for (int i = 0; i < fData.size(); i++)
    {
        fData[i] = rand() % 1001;
    }
    
    for (int j = 0; j < fData.size(); j++)
    {
        fMessage.add_samples(fData[j]);    
    }

    return 0;
}

But I want to use a method like memcpy to accelerate the copy process. It is just an idea that comes to my mind. If it's completely wrong correct me. The last declaration in the headerfile is:

inline ::google::protobuf::RepeatedField< float >* mutable_samples();

I have no idea what this method does (lack of skill). But it kind of looks like a vector. Maybe that's the solution for my problem. If so, I have no idea how to implement it.

Have you profiled it, and seen that it's slow? Have you inspected the compiled code, and seen that it's poorly optimized? — Useless, Mar 19 '13 at 13:23

score 56 · Accepted Answer · answered Apr 13 '18 at 17:45

56

Since this isn't here yet and I like one-liners:

*fMessage.mutable_samples() = {fData.begin(), fData.end()};

answered Apr 13 '18 at 17:45

mgild

774
6
13

1

Note that it works with c++11 and later, because reassignment using initializer list was not available before c++11. – Siddu Jan 03 '19 at 12:32
Does this assignment copy, or move the used memory? – Dávid Tóth Sep 19 '19 at 08:18
1

@DavidTóth it copies the range into a temporary and then the temporary is move assigned to the pbf repeated field – Kevin Kreiser Oct 23 '20 at 13:08
I would greatly appreciate if someone linked to actual documentation as I don't even have keywords to search. I am talking about this two-iterator braces initializer. – hamilyon Oct 27 '20 at 06:48
2

@hamilyon https://en.cppreference.com/w/cpp/language/list_initialization – mgild Oct 29 '20 at 04:06
But how do you know `RepeatedField` is `std::vector` behind the scene? `list_initialization` is something for standard library stuffs. – Rick Nov 17 '20 at 10:18
5

Note that the reverse is also possible with the same one liner: std::vector samples = {fMessage.samples().begin(), fMessage.samples().end()}; – Wagner Volanin Jun 09 '21 at 22:26
1

@Rick If a list_initialization constructor is not found, it will try to find a constructor that fits the data values in the list. In this case, there is a template constructor `RepeatedField(Iter begin, Iter end)` which is used. So no need to use `std::vector` behind the scenes for it to work. – Azmisov Jul 14 '21 at 20:56

score 23 · Answer 2 · edited Oct 21 '20 at 16:32

23

I found the shortest way to copy vector into repeated field as this:

google::protobuf::RepeatedField<float> data(fData.begin(), fData.end());
fMessage.mutable_samples()->Swap(&data);

It is probably also faster than yours since it avoids initial iteration and setting values to 0.

edited Oct 21 '20 at 16:32

ndmeiri

4,979
12
37
45

answered Mar 03 '17 at 06:15

nazgul

449
1
4
10

score 2 · Answer 3 · answered Jul 14 '21 at 20:51

@mgild's answer will implicitly call the RepeatedField(Iter begin, Iter end) constructor to create a temporary to be move assigned. Same with @nazgul's answer, which explicitly creates a temporary RepeatedField and swaps it.

Much simpler, avoiding the creation of a new object would be:

fMessage.mutable_samples()->Add(fData.begin(), fData.end())

If samples field is not empty already, you can call Clear method prior.

Internally, this uses std::copy (so long as fData is a forward iterator), so will be just as fast as any memcpy implementation you come up with.

Dávid Tóth · Answer 4 · 2022-03-15T13:01:57.133

1

As an alternative to the excellent answer posted by @mgild, Assign can also be used in this situation, so the data would not be copied to a temporary before being moved.

fMessage.mutable_samples()->Assign(fData.begin(), fData.end())

Credit to Marek R's answer

edited Mar 15 '22 at 13:01

answered Mar 15 '22 at 12:51

Dávid Tóth

2,788
1
21
46

score -3 · Answer 5 · answered Sep 12 '19 at 09:08

-3

fMessage.mutable_samples()

return an array of pointer of samples : [*sample1, *sample2, sample3, ...].

&fData[0]

is the address of first element of fData.

memcpy(fMessage.mutable_samples()->mutable_data(),
     &fData[0],
     sizeof(float)*fData.size());

So I do not think the code above can successfully fill data from fData to fMessage. It's totally wrong!

answered Sep 12 '19 at 09:08

Arthur George

43
6

No it doesn't. `mutable_samples()` points to a `RepeatedField`, and `RepeatedField::mutable_data` returns `float*`. Have [some documentation](https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.repeated_field#RepeatedField.mutable_data.details). – Asteroids With Wings Oct 21 '20 at 16:37

Copy a std::vector to a repeated field from protobuf with memcpy

5 Answers5

Linked