1

I need a way to write structures of three different kinds to a binary file, which later has to be searched. (As in, for example, struct A has two fields, an int and a char; struct B has int and a long; I need to output all structures whose int equals the one given from keyboard).

I understand how to write structs of the same kind to a file and how to search them, but here I am just lost, best thing I came up with is declaring a struct containing all possibly needed fields and leaving the ones I don't need empty, but it really feels wrong, there HAS to be a better way to do that.

I've read about binary files and could not find anything relevant, most examples and tutorials deal with writing one data type. Could anyone point me in the right direction?

EDIT: I am looking for what @Jerry_coffin called database mode, and will probably use one of the existing database systems for that, best way to go, probably. Thank you everybody for the suggestions

pchr8
  • 115
  • 7
  • you can have a first member of type int indicating what kinf of struct it is – 4pie0 May 29 '14 at 23:54
  • Use a sensible serialization scheme, e.g. protocol buffers or Boost. – Kerrek SB May 29 '14 at 23:55
  • @bits_international that would make sense, but when reading/writing I have to use sizeof(struct whatever), so in this case I have to make assumptions about the size of the structure first, and actually learn what kind of structure it is later? – pchr8 May 30 '14 at 00:02
  • If you want fast random access, leaving some "fields" blank depending on type may in fact be the way to go. – aschepler May 30 '14 at 00:03
  • You could use a union. Or if you have all the data before you write it, sort it by type then id when you write it. You can then find in 3 quick binary searches – AShelly May 30 '14 at 00:08
  • You can do like @bits_international said. When reading, read the type first then you'll know how many remaining bytes you have to read more – phuclv May 30 '14 at 00:29
  • @user58053 that's not a great idea as `sizeof(struct whatever)` may change depending on your compiler flags, or if you compile on a different system – M.M May 30 '14 at 00:34

2 Answers2

2

There are two basic ways to deal with your data (and you haven't said which applies):

  1. What I think of as the word processor model: read a file in, use/modify the data as needed, write all the data back out when the user saves (and maybe automatically, even if the user doesn't explicitly ask for a save).
  2. What I think of as a the database mode: you don't normally plan on reading an entire file into memory at once. You update pieces and parts of the file as needed. You structure the file to support more or less random access.

If you're planning on the first, things are pretty simple. Keep your three structures separate in memory. When you write them out, you write all of one type, then all of the next, and finally all of the third. You need enough metadata somewhere to be able to read the data back in correctly.

If you're planning on the second, it's probably more complex. You have a few different possibilities along this line:

  1. Divide the data into three separate files and search each one separately. I know you said you need a single file, but if you can do this, it's probably the cleanest approach.
  2. In your source code, create a union of the three types, along with a field to tell which type the rest is. This is basically just leaving the space unneeded by a particular struct empty, but making it slightly simpler to map between the external and internal representations.
  3. Build something closer to a full-blown database system. Basically, you'll want to write the raw records into the file. Then you'll want some sort of index to tell where each record starts. Then you may want to add at least one index to help find particular records quickly and easily.
  4. Use an existing database system. SQLite is one obvious choice. Another that's not as well known is STXXL. Obviously enough, SQLite provides a SQL interface to the data, which makes it much easier to support things like ad-hoc queries and such (if needed). STXXL provides an interface similar to the containers in the standard library, but with the data stored on disk. This has been pretty heavily tested and optimized; it'll typically take quite a bit of effort to match its performance.

As to which is preferable: it depends. Separate files is probably the simplest. Given specific requirements subject to minimal change, I'd probably design a single type as suggested in 2, and if I needed to store a lot of data, probably use STXXL. If queries you might need are much more fluid, I'd probably consider SQLite. I'd avoid writing your own database system unless your needs are extremely specific and unusual, so you see a strong likelihood of minimal effort leading to a major performance improvement for your use. Frankly, I think that's pretty unlikely though.

Jerry Coffin
  • 476,176
  • 80
  • 629
  • 1,111
  • I actually meant database mode, sorry for not making my question clear. The fourth variant probably is the best for me, will look into SQLite or STXXL. Thank you! – pchr8 May 31 '14 at 15:20
0

I would suggest to use protobuf

Say you create following protobuf message, for example.

message binary_info
{
    message struct_1
    {
        required int32 intValue = 1;
        required string strValue = 2;
    }

    message struct_2
    {
        required int32 intValue = 1;
        required uint64 ulongValue = 2;
    }

    message struct_3
    {
        required int32 intValue = 1;
        required uint64 ulongValue = 2;
        required string strValue = 3;
    }

    required struct_1 st1 = 1;
    required struct_2 st2 = 2;
    required struct_3 st3 = 3;
}

Let google protobuf create necessary files for you. See developer guide at https://developers.google.com/protocol-buffers/docs/overview

(In my example, protoc has auto generated message.pb.h and message.pb.cc files at communications/proto/*)

You can then write a program to save and read data to/from file

/*
 * main.cpp
 *
 *  Created on: 30/05/2014
 *      Author: ankit
 */

#include "communications/proto/message.pb.h"
#include <iostream>
#include <fstream>


using namespace std;

bool write_data()
{
    binary_info_struct_1* s1 = new binary_info_struct_1();
    s1->set_intvalue(1);
    s1->set_strvalue("string for s1");

    binary_info_struct_2* s2 = new binary_info_struct_2();
    s2->set_intvalue(2);
    s2->set_ulongvalue(2000);

    binary_info_struct_3* s3 = new binary_info_struct_3();
    s3->set_intvalue(3);
    s3->set_ulongvalue(3000);
    s3->set_strvalue("string for s3");

    binary_info b;

    b.set_allocated_st1(s1);
    b.set_allocated_st2(s2);
    b.set_allocated_st3(s3);

    if(!b.IsInitialized())
        return false;

    fstream output("myfile.data", ios::out | ios::binary);
    b.SerializeToOstream(&output);

    return true;
}

void read_data()
{
    fstream input("myfile.data", ios::in | ios::binary);

    binary_info b;
    b.ParseFromIstream(&input);

    cout << "struct 1, int data: " << b.st1().intvalue() << endl;
    cout << "struct 1, string data: " << b.st1().strvalue() << endl;

    cout << "struct 2, int data: " << b.st2().intvalue() << endl;
    cout << "struct 2, ulong data: " << b.st2().ulongvalue() << endl;

    cout << "struct 3, int data: " << b.st3().intvalue() << endl;
    cout << "struct 3, ulong data: " << b.st3().ulongvalue() << endl;
    cout << "struct 3, string data: " << b.st3().strvalue() << endl;
}

int main()
{
    write_data();
    read_data();
    return 0;
}

Hope this helps!

Ankit Patel
  • 1,191
  • 1
  • 9
  • 9