Streaming data with ProtoBuf to log file with a header

Question

I am trying to stream data to a log file on an SD card on a microcontroller that reads from some sensors and stores the value in the file.

To serialize the data I will be using NanoPB, the protobuf implementation for C which is quite resource efficient.

The log file has the following structure: It needs to write a short header composed of a GUID, and a firmware version. After the header, the stream of data should be continuous and it should log the fields from the sensors but not the header values (this should only occur once and at the beginning).

The limitation is that I can only use one .proto file to serialize and deserialize and I want to avoid Pb_callback functions that emerge from using "repeated" fields in the .proto and then using the C implementation of nanopb. https://jpa.kapsi.fi/nanopb/docs/concepts.html .

The implementation i've tried is the following (fields are just examples):

syntax = "proto3";

import "timestamp.proto";
import "nanopb.proto";

message LogHeader {
    string firmware = 1 [(nanopb).max_size = 11];  
    string GUID = 2 [(nanopb).max_size = 11];       
}

message Sensors {
    int32 TimeStamp = 3;        
    // Sensory data
    int32 Sens1 = 4;
    int32 Sens2 = 5;
    int32 Sens3 = 6;
    int32 Sens4 = 7;
    int32 Sense5 = 8;

}

The idea would be to have a log file that would look like this once processed:

firmware "1.0.0"
GUID "1231214211321" (example)
Timestamp 123123
Sens1 2343
Sens2 13123
Sens3 13443
Sens4 1231
Sens5 190
Timestamp 123124
Sens1 2345
Sens2 2312
...

But if all fields were to be in the same message the GUID and firmware will be logged every repetition. While if I split it in 2 messages I have not been able to deserialize them in one go with one proto file. I would need to know the length of the first two messages, deserialize them and then start from there on with the log.

score 1 · Accepted Answer · answered Oct 23 '19 at 08:32

I want to avoid Pb_callback functions that emerge from using "repeated" fields in the .proto

Note that you can specify max_count for repeated fields just like you've specified max_size for strings, and then you'll get a simple array instead of a callback.

While if I split it in 2 messages I have not been able to deserialize them in one go with one proto file.

Protobuf deserialization needs to know the message type. The most common way to handle this is to have a single top-level message with submessages:

message LogMessage {
   optional LogHeader header = 1;
   optional Sensors sensors = 2;
}

Then you can set either or both of the header and sensor fields, and has_header and has_sensors to true or false to indicate whether you want to include that subfield. But no matter the contents, you always serialize and deserialize as LogMessage, so there is no confusion between the different message types.

I would need to know the length of the first two messages, deserialize them and then start from there on with the log.

Yeah, that is a common beginner problem with protobuf also. Protobuf messages themselves do not encode their length, so if you have multiple messages in a single file you need to separate them somehow.

A quite common way is to add a length prefix, as done by nanopb's pb_encode_delimited() and pb_decode_delimited(). This format is also supported by the C++ protobuf library. However, a drawback of that is that many command line tools like protoc do not support the delimited format, and e.g. Python protobuf library makes decoding them somewhat complex.

Another option is to make the whole file appear like it was a single message, but write it in multiple parts. Protobufs have the merge feature, that is, if you just append messages after one another, they'll merge together. This can be done by including repeated fields in the LogMessage:

message LogMessage {
   optional LogHeader header = 1;
   repeated Sensors sensors = 2 [(nanopb).max_count = 1];
}

Now if you encode multiple copies of the LogMessage, each with a single sensors entry, they'll merge together. Then if you decode the file it will appear like a single LogMessage with multiple sensors entries.

If I understood correctly, isn't .max_count refer to the maximum number of repetitions you expect (which I don't know) ? I don't understand why you set it to 1. I know it is to avoid the callback, but why to 1? Thanks for the explanation @jpa. — Asan, Oct 23 '19 at 08:46
@A.San If you only want to encode one sensors submessage at a time, 1 is enough. But yeah, you won't be able to decode the combined message that way with nanopb, you'd need callbacks for that. — jpa, Oct 23 '19 at 08:54
Ok, In this case I just need to encode with nanopb and decode elsewhere. So this should do the job of encoding. — Asan, Oct 23 '19 at 09:01

Streaming data with ProtoBuf to log file with a header

1 Answers1

Linked