Porting data serialization code from C++ linux/mac to C++ windows

Question

I have a software framework compiled and running successfully on both mac and linux. I am now trying to port it to windows (using mingw). So far, I have the software compiling and running under windows but its inevitably buggy. In particular, I have an issue with reading data that was serialized in macos (or linux) into the windows version of the program (segfaults).

The serialization process serializes values of primitive variables (longs, ints, doubles etc.) to disk.

This is the code I am using:

#include <iostream>
#include <fstream>

template <class T>
void serializeVariable(T var, std::ofstream &outFile)
{
    outFile.write (reinterpret_cast < char *>(&var),sizeof (var));
}

template <class T>
void readSerializedVariable(T &var, std::ifstream &inFile)
{
inFile.read (reinterpret_cast < char *>(&var),sizeof (var));
}

So to save the state of a bunch of variables, I call serializeVariable for each variable in turn. Then to read the data back in, calls are made to readSerializedVariable in the same order in which they were saved. For example to save:

::serializeVariable<float>(spreadx,outFile);
::serializeVariable<int>(objectDensity,outFile);
::serializeVariable<int>(popSize,outFile);

And to read:

::readSerializedVariable<float>(spreadx,inFile);
::readSerializedVariable<int>(objectDensity,inFile);
::readSerializedVariable<int>(popSize,inFile);

But in windows, this reading of serialized data is failing. I am guessing that windows serializes data a little differently. I wonder if there is a way in which I could modify the above code so that data saved on any platform can be read on any other platform...any ideas?

Cheers,

Ben.

Answer 1

this is just a wild guess sry I can't help you more. My idea is that the byte order is different: big endian vs little endian. So anything larger than one byte will be messed up when loaded on a machine that has the order reversed.

For example I found this peace of code in msdn:

int isLittleEndian() {
    long int testInt = 0x12345678;
    char *pMem;

    pMem = (char *) testInt;
    if (pMem[0] == 0x78)
        return(1);
    else
        return(0);
}

I guess you will have different results on linux vs windows. Best case would be if there is a flag option for your compiler(s) to use one format or the other. Just set it to be the same on all machines.

Hope this helps, Alex

Answer 2

Binary serialization like this should work fine across those platforms. You do have to honor endianness, but that is trivial. I don't think these three platforms have any conflicts in this respect.

You really can't use as loose of type specifications when you do, though. int, float, size_t sizes can all change across platforms.

For integer types, use the strict sized types found in the cstdint header. uint32_t, int32_t, etc. Windows doesn't have the header available iirc, but you can use boost/cstdint.hpp instead.

Floating point should work as most compilers follow the same IEEE specs.

C - Serialization of the floating point numbers (floats, doubles)

Binary serialization really needs thorough unit testing. I would strongly recommend investing the time.

Answer 3

Just one more wild guess: you forget open file in binary reading mode, and on windows file streams convert sequence 13,10 to 10.

Answer 4

Did you consider using serialization libraries or formats, like e.g.:

• XDR (supported by libc) or ASN1
• s11n (a C++ serialization library)
• Json, a very simple textual format with many libraries for it, e.g. JsonCpp, Jansson, Jaula, ....)
• YAML, a more powerful textual format, with many libraries
• or even XML, which is often used for serialization purposes...

(And for serialization of scalars, the htonl and companion routines should help)