How can I use fread() on a binary file to read the data into a std::vector?

Question

Follow-up question on an earlier question I had, that has been perfectly answered. To quickly recap, I had trouble creating a class holding a huge array (stack overflow error). In the answers, some users recommended I use std::vector instead.

The function to read in the data looks like this:

Test()
{
   memset(myarray, 0, sizeof(myarray));
   FILE* fstr = fopen("myfile.dat", "rb");
   size_t success= fread(myarray, sizeof(myarray), 1, fstr);
   fclose(fstr);
}

for a myarray which looked like this:

int myarray[45000000];

My question is: How can I read this into a preferable:

std::vector<int> myvector;

I searched google , and have found multiple answers, usually pointing to the following code:

std::ifstream input("myfile.dat", std::ios::in | std::ifstream::binary);
std::copy(std::istream_iterator<int>(input), 
     std::istream_iterator<int>(), 
     std::back_inserter(myvector));

After implementing this, and when calling myvector.size() I get 16 (for whatever reason), and accessing a vector element leads to an immediate crash for going out of the vector bounds.

So what do I have to do to get this right? I once read somewhere that I could just simply use the "old" method, and then reading the array into the vector, but this seems to defeat the purpose of using the vector in the first place.

`istream_iterator` does formatted input. If your file is in binary format, that's not what you want. — jrok, Feb 28 '13 at 19:11
@MichaelDorgan: Way to not read the question at all. @MarkAnderson: `std::istream_iterator` does _formatted_ reads, and the old `fread` is an _unformatted_ read. You need to do an unformatted read into the vector. — Mooing Duck, Feb 28 '13 at 19:11

Zeta · Accepted Answer · 2013-02-28T19:17:29.433

6

fread() reads your file binary, while ifstream_iterator tries to extract formatted ints (like 42).

You want to resize your vector and use input.read(...) instead:

const size_t size = 45000000; // change this to the appropriate value
std::vector<char> myvector(size, 0);
std::ifstream input("myfile.dat", std::ios::in | std::ifstream::binary);
input.read(&myvector[0], myvector.size());

Note that you need to use a std::vector<char> since read expects the first parameter to be a char *. You can use other types T if you cast the type correctly:

input.read(reinterpret_cast<char*>(&myvector[0]), myvector.size() * sizeof(T));

edited Feb 28 '13 at 19:17

answered Feb 28 '13 at 19:11

Zeta

103,620
13
194
236

Would you need to call free() on &myvector[0] after you're done? – Ol1v3r Jan 19 '21 at 10:47
2

@OliverCiappara No. You *never* want to call `free` on `&myvector[0]` or `myvector.data()` for any `std::vector`. `std::vector` handles allocation and deallocation on its own. Remember: if you didn't call `malloc`, you must not call `free` (unless the function's documentation tells you to use the latter, e.g. `strdup` in POSIX). – Zeta Jan 19 '21 at 11:12

score 4 · Answer 2 · answered Feb 28 '13 at 19:17

If you're using C++ you should try to avoid using the C FILE APIs all together -- so you're on the right track. The problem you're having is that istream_iterator reads input as text, not binary -- it's expecting ASCII digits. This this out instead:

std::vector<int> vec(45000000);

std::filebuf fb;
fb.open("myfile.dat", std::ios_base::in | std::ios_base::binary);
fb.sgetn((char*)&vec[0], vec.size() * sizeof(vec[0]));

How can I use fread() on a binary file to read the data into a std::vector?

2 Answers2