3

In my current project I have a few different interfaces that require me to serialize messages into byte buffers. I feel like I'm probably not doing it in a way that would make a true C++ programmer happy (and I'd like to).

I would typically do something like this:

struct MyStruct {
    uint32_t x;
    uint64_t y;
    uint8_t  z[80];
};

uint8_t* serialize(const MyStruct& s) {
    uint8_t* buffer = new uint8_t[sizeof(s)];
    uint8_t* temp = buffer;
    memcpy(temp, &s.x, sizeof(s.x));
    temp += sizeof(s.x);

    //would also have put in network byte order...
    ... etc ...

    return buffer;
}

Excuse any typos, that was just an example off the top of my head. Obviously it can get more complex if the structure I'm serializing has internal pointers.

So, I have two questions that are closely related:

  1. Is there any problem in the specific scenario above with serializing by casting the struct directly to a char buffer assuming I know that the destination systems are in the same endianness?

  2. Main question: Is there a better... erm... C++? way to do this aside from the addition of smart pointers? I feel like this is such a common problem that the STL probably handles it - and if it doesn't, I'm sure there's a better way to do it anyway using C++ mechanisms.

EDIT Bonus points if you can do a clean example of serializing this structure in a better way using standard C++/STL without added libraries.

John Humphreys
  • 37,047
  • 37
  • 155
  • 255
  • Google has a [Protocol Buffers](http://code.google.com/apis/protocolbuffers/docs/overview.html) – Joe Nov 15 '11 at 15:05

3 Answers3

6

You should probable use either Boost::serialization or streams directly. More information in either of the links to the right.

Is it possible to serialize and deserialize a class in C++?

Community
  • 1
  • 1
AzP
  • 1,081
  • 1
  • 16
  • 27
  • 1
    +1 because this looks interesting - but if you can add an answer that neatly does it with native C++ streams/STL I'll definitely accept - I work on an embedded platform and the process to get approval for adding libraries is pretty tough :( – John Humphreys Nov 15 '11 at 15:15
  • Well I _could_ add an answer that does it, but I just thought "the links to the right" had so much information I wouldn't reproduce it here :) – AzP Nov 17 '11 at 13:00
6

You might want to take a look at Google Protocol Buffers (also known as protobuf). You define your data in a language neutral IDL and then run it through a generator to generate your C++ classes. It'll take care of byte ordering issues and can provide a very compact binary form.

By using this you'll not only be able to save your C++ data but it'll be useable in other languages (C#, Java, Python etc) as there are protobuf implementation available for them.

Sean
  • 60,939
  • 11
  • 97
  • 136
  • 1
    +1 because this looks interesting - but if you can add an answer that neatly does it with native C++ streams/STL I'll definitely accept - I work on an embedded platform and the process to get approval for adding libraries is pretty tough :( – John Humphreys Nov 15 '11 at 15:16
1

just voted AzPs reply as the answer, checking Boost first is the way to go.

in addition about your code sample:

1 - changing the signature of your serialization function to a method taking a file:

void MyStruct::serialize(FILE* file) // or stream
{
    int size = sizeof(this);
    fwrite(&size, sizeof(int), 1, file); // write size
    fwrite(this, 1, size, file);         // write raw bytes of struct
}

reduces the necessity to copy the struct.

2 - yes, your code makes the serialized bytes dependent on your platform, compiler and compiler settings. this is not good or bad, if the same binary writes and reads the serialized bytes, this might be beneificial because of simplicity and performance. But it is not only endianness, also packing and struct layout affects compatibility. For instance a 32bit or a 64bit version of your app will change the layout of our struct quite for sure. Finally serializing the raw footprint also serializes padding bytes - the bytes the compiler might put between struct fields, an overhead undesirable for high traffic network streams (see google protocol buffers as they hunt every bit they can save).

EDIT:

i see you added "embedded". yes, then such simple serialize / deserialize (mirror implementation of the above serialize) methods might be a good and simple choice.

Christian Rau
  • 45,360
  • 10
  • 108
  • 185
citykid
  • 9,916
  • 10
  • 55
  • 91
  • And stdio's `FILE` pointers are a C++ way of serialization? I know you mentioned a stream in the comment, but it doesn't belong into a comment, but into the code! Otherwise the idea of passing the file or stream to the function is of course a better idea than allocating a new buffer in each function invocation. – Christian Rau Nov 15 '11 at 16:46
  • Well, I get your point. The c++ way would be a std::ostream, yes. But in particular the std::stream libraries often come with surprises and there is nothing wrong going down to the raw c API, in particular the quite platform neutral file api. Here, avoiding std::stream frees for instance from thinking about thhose libraries internal buffering and flushing. If its really just saving structs in a file and I do not really know the internals of std::stream I have to deal with, raw files are just pragmatic, simple and surely faster than std::stream. – citykid Nov 15 '11 at 17:37
  • @thomas Can't stdio `FILE`s use buffering, too? And I'm not sure if `fwrite` is really any faster than `ostream::write`, maybe by a function call, which will be outruled by the writing itself, anyway. Of course you don't use `operator<<` for binary serialization. I just don't see how `out.write(...)` would be any different from `fwrite(FILE, ...)`, except for being the C++ way, which the OP asked for. – Christian Rau Nov 15 '11 at 18:33
  • well, the "c++ way" as shown in many text books and what "true c++ programmers" do is not the same. I am not for a side show discussion here, should suffice to say that many if not most great c++ shops limit iostreams. As we talked here protocol buffers, grab the C++ source and find FILE* there. What I do agree is that FILE* looks C-ish therefore I personally carefully use int as file handles, and for *nix, windows compatility this is ok. Yes, many or most knowledgeable C++ programmers do binary file or socket accesss at stdio level, for many reasons. And I left room for the others ... – citykid Nov 23 '11 at 16:44