Save and load data using ANSI C on any platform

Question

Say I have 1 million structs, each containing integers, doubles, strings, and other structs, something like:

struct s1 {
    int f1;
    long f2;
    char* f3;
};

struct s2 {
    struct s1* f1;
    double f2;
};

How can I save these to a file in binary format, then look up and load them from that file, on platforms different from the one the executable was compiled on, without worrying about endianness, float representation and other platform-specific gotchas?

The reason for preferring a binary format is mainly size of the resulting file. If integers alone look like "32435" and I have millions of them, the extra 3 bytes per integer would add quite a bit of size to the file.

Disk storage is ridiculously cheap - why do you care? Also, note that the number "1" as text takes up a lot less space than it's binary representation as an `int`, so these things tend to even out. — , Jun 11 '11 at 18:00
possible duplicate of [Serialize Data Structures in C](http://stackoverflow.com/questions/371371/serialize-data-structures-in-c) — dmckee --- ex-moderator kitten, Jun 11 '11 at 18:03
@dmckee, the poster for that question does not care about data portability, whereas portability is my main concern. — rid, Jun 11 '11 at 18:05
@Neil, @Radu: and if it doesn't even out, there's always gzip. — Steve Jessop, Jun 11 '11 at 18:19

score 3 · Answer 1 · answered Jun 11 '11 at 17:37

3

Write them as ascii text, XML or some similar non-binary format.

answered Jun 11 '11 at 17:37

score 2 · Accepted Answer · answered Jun 11 '11 at 18:11

"platforms different from the one the executable was compiled on"

How different from the one the executable was compiled on? Do you need to support platforms that use non-IEEE floats? Platforms that use non-ASCII characters? Platforms that use non-8-bit bytes?

If you insist on binary, and insist on doing it yourself, probably your best bet is to define that in the storage format, int and long will each be stored as a sequence of 4 bytes, little-endian (or big-endian, but pick one and stick to it regardless of platform), containing exactly 8 significant bits per byte. double will be an IEEE double likewise. Pointers introduce a whole world of hurt, the storage format must attach a unique identifier to each instance of s1, and then a pointer to s1 can be stored as an id value, and looked up as part of deserialization.

Different platforms can then decide what types they want to use for each of the storage types (so for example if int is only 16 bits on a give platform, it will just have to use long for both the int and long types. For this reason, you should give them domain-specific pseudonyms). Beware that it's impossible to avoid loss of precision in double values when converting to and from incompatible representations, since they might not have the same number of significant bits.

For text, non-ASCII platforms will have to include code to serialize their own text format to ASCII, and to deserialize ASCII to native text. Strictly speaking, you should also avoid using any characters in the text that aren't in the C basic character set, since they might not be representable at all on the target. You can make a similar decision whether you're willing to count on target platforms to support Unicode in some way -- if so then UTF-8 is a reasonable interchange format for text.

Finally, for each struct on each platform, you can either:

write (or perhaps auto-generate) code to serialize it, and code to deserialize it, or:
make yourself a domain-specific language to define the structures, and a parser/interpreter that will serialize and deserialize according to that definition.

Sounds like a lot of work to me, though, to do something that's been done before.

That's a very good answer, thank you. I'm not insisting on doing it myself, I'm just looking for solutions. If this was done before in a ANSI C library that I can use, I have no problem with it. — rid, Jun 11 '11 at 18:16
@Radu: there's http://code.google.com/p/protobuf-c/, or you could look into ASN.1 libraries. I don't much like ASN.1, but it exists and it's widely used. I don't think either of them completely deals with pointers, you'll have to build on them for that, but they'll do the rest for any platform that isn't implausibly weird. — Steve Jessop, Jun 11 '11 at 18:18

score 0 · Answer 3 · answered Jun 11 '11 at 17:42

0

If you want to avoid the headaches that you've described, DON'T use binary. Use text, the universal* format.

*until you start getting into locales.

answered Jun 11 '11 at 17:42

Chris Eberle

47,994
12
82
119

Save and load data using ANSI C on any platform

3 Answers3