How to read and write many objects (or any data) to a buffer?

Question

I am writing a program to save some object (struct) to a buffer. I don't have experiment to write many objects to buffer and read these objects from the buffer. Any help would be appreciated. My code can write one item to object and I want write many objects to the buffer

struct PointFull {
    double lat;
    double lon;
};

PointFull item1;
PointFull item2;
PointFull item3;
PointFull item4;
PointFull item5;

void* buffer = malloc(sizeof (PointFull));
int fd = open("output", O_WRONLY | O_CREAT, S_IWUSR | S_IRUSR);
if (fd < 0) {
    printf("Error opening file\n");
    return 1;
}

//The below function can write only one item to buffer. How to write 5 item (from item1 to item5) to the buffer
//memcpy(buffer, &item1, sizeof (item));

write(fd, buffer, sizeof (item));

Now I have a file named "output" in hard disk and then I want read the file to test data.

    int fd2 = open("output", O_RDONLY, S_IWUSR | S_IRUSR);
if (fd2 < 0) {
    printf("Error opening file\n");
    return 1;
}
void* bufferRead;
bufferRead = malloc(5* sizeof (PointFull));
read(fd2, bufferRead,sizeof (PointFull));

At the moment, I have bufferRead contain 5 items but I dont know how to read buffer to insert data to struct??? Plz help me!

In your first example, how about using an array, like `PointFull items[5]`? And if you want to use C++, then use `new`, not `malloc`! — Mats Petersson, Aug 27 '15 at 07:39
@MatsPetersson: Thanks for you reply. In my situation, I want to write one object and after a long time I need write other object so I can not use array. I want write objects in turn. Do you have any idea :) — Mozart, Aug 27 '15 at 08:07
If you are using C++, you could use boost serialization. Look for boost/archive in the following link for a simple example: http://stackoverflow.com/questions/523872/how-do-you-serialize-an-object-in-c — Erik Alapää, Aug 27 '15 at 08:23

score 3 · Accepted Answer · edited May 23 '17 at 12:07

Well what you want to do is serialization. Say you have such structure:

struct PointFull {
    int lat;
    int lon;
};

and also

PointFull item1, item2;

The way you serialize it to buffer is:

unsigned char arr[20] = {0};
memcpy(arr, &item1.lat, sizeof(int));
memcpy(&arr[1 * sizeof(int)], &item1.lon, sizeof(int));
memcpy(&arr[2 * sizeof(int)], &item2.lat, sizeof(int));
memcpy(&arr[3 * sizeof(int)], &item2.lon, sizeof(int));

I am serializing like this because it is not good idea to directly write the structure like you suggest because of padding problems. The structures may have paddings and they may differ per system.

Now, you have the byte array (which contains two PointFull objects - for more objects you would follow similar approach) and you can use it in your write:

write(fd, arr, 20);

After reading the byte array you could reconstruct point objects by using similar memcpy calls like above (just destination would be the point object members now). But the problem with this is that integer serialization in binary is not portable (and floats moreover) - on different systems integers may have different size, different endianness. With floats additionally their representation may differ.

Anyway there is one way how to encode floats in binary here - check pack754 (and similar unpack) function. If you use that function for serializing floats in byte array and serialize each float separately like in this answer, then maybe you will be fine.

_{PS. here is post which explains serialization (for float encoding part you can use link in my answer).}

Thanks for your reply. I have a question, why do you have to write lon lat in arr[]? Can I write whole item to arr[]? memcpy(&arr[1 * sizeof(PointFull)], &item1, sizeof(PointFull)); — Mozart, Aug 27 '15 at 09:02
@MaiNhậtTân: I already mentioned - because of padding problems. Search for structure padding — Giorgi Moniava, Aug 27 '15 at 09:07
I got it. Now, I have file, how can I read items from the file. Using each buffer for one item? — Mozart, Aug 27 '15 at 09:58
@MaiNhậtTân: Check also link I gave. You read similarly. Read the whole file to byte array. Then from array do memcpy to each object's members like `memcpy(&item1.lat, arr, sizeof(int));` etc. Similarly like you did with write, just with read you change order in memcpy — Giorgi Moniava, Aug 27 '15 at 10:01

score 0 · Answer 2 · answered Aug 27 '15 at 08:37

If you just want to write structures to a file, you can just write them directly :

write(fd, &item1, sizeof (item1));
write(fd, &item2, sizeof (item2));
...

Unless you really have a lot of those, it will perform alright, the OS itself will buffer the file system access.

If you really want to use a buffer, you can have a small class to write to the buffer :

class BufferWriter {
    char *ptr;
    int current;
public:
    BufferWriter( void *ptr ) : ptr((char*)ptr), current(0) {}
    template<typename T>
    void Write( const T &t ) {
        memcpy(&ptr[current], &t, sizeof(t) );
        current += sizeof(t);
    }
    int GetTotalLength() {
        return current;
    }
};

And then use it like that :

char *b = new char[ENOUGH];
BufferWriter buffer(b);

buffer.Write(item1);
buffer.Write(item2);
...

write(fd, b, buffer.GetTotalLength());
delete[] b;

You can add some code to check for buffer overflow if you want.

About the portability of the file you output (if you plan to transfer it to another device), you may want to use intX_t (ex: int32_t) instead of int, short or whatever to be sure of the size. Also for ints, you may have to check the endianness of the system, but on personal devices it will always be little-endian. You don't have to worry about float and double, because all modern devices use the IEEE 754 norm, and even most of exotic devices also stick to this.

How to read and write many objects (or any data) to a buffer?

2 Answers2