Getting desired binary data ranges from std::istreambuf_iterator and std::ifstream

Question

#include <iostream>
#include <fstream>
#include <iterator>
#include <vector>

int main()
{
    std::ifstream file("data.bin", std::ios::binary );

    if( file.fail() )
    {
        std::cout << "File does not exist or could not open file";

        return 0;
    }

    std::vector<short> buffer;

    std::copy( 
            std::istreambuf_iterator<char>( file ),
            std::istreambuf_iterator<char>(),
            std::back_inserter( buffer )
            );

   return 0;
}

This only gives me ranges of char values (-128, 128).

I thought using istreambuf_iterator<short> would give me what I want but it throws an "invalid conversion" error.

What can I do to read binary values that are in the short range?

@DeadMG It's only one, and I did mention it was an "invalid conversion" error. — Tek, Dec 02 '12 at 13:29

score 2 · Answer 1 · answered Dec 02 '12 at 01:32

The class std::istreambuf_iterator<cT> iterates over the characters extracted from a std::basic_istream<cT, std::char_traits<cT>> or, actually, its std::basic_streambuf<cT, std::char_traits<cT>>. That is, given a specific stream type, there is no choice for the std::istreambuf_iterator<cT>. Also note that the IOStream layer is not intended to operate on binary files, despite the std::ios_base::binary operation which is passed to the stream buffers.

If you want to extract shorts (or any other type than the stream's character type) you'd need to create a suitable iterator yourself which takes the chosen binary format the file was written in into account. This iterator can be built in terms of std::istreambuf_iterator<char> or directly access the std::streambuf.

The explanation is great, thanks. But code examples would be much more appreciated to show what you mean. I'm new to C++ and I'm sure other people that will read this too. You don't have to of course, just a suggestion. — Tek, Dec 02 '12 at 01:47

sehe · Accepted Answer · 2012-12-02T01:38:01.130

1

Here's my recommended Spirit approach:

#include <fstream>
#include <boost/spirit/include/qi.hpp>

namespace qi = boost::spirit::qi;

int main()
{
    std::ifstream file("data.bin", std::ios::binary);
    if(!file)
        std::cout << "File does not exist or could not open file";
    boost::spirit::istream_iterator f(file), l;

    std::vector<int16_t> buffer;
    bool ok = qi::parse(f, l, *qi::word, buffer);
}

Of course, there is qi::dword, qi::qword, big-endian/little-endian variatations etc.:

You can look at wchar_t

#include <iostream>
#include <fstream>
#include <iterator>
#include <vector>
int main()
{
    std::wifstream file("data.bin", std::ios::binary );
    if( file.fail() )
    {
        std::cout << "File does not exist or could not open file";
        return 0;
    }
    std::vector<wchar_t> buffer;
    std::copy( 
            std::istreambuf_iterator<wchar_t>( file ),
            std::istreambuf_iterator<wchar_t>(),
            std::back_inserter( buffer )
            );
}

I'm not too sure whether the actual size of wchar_t is implementation defined.

edited Dec 02 '12 at 01:38

answered Dec 02 '12 at 01:32

sehe

374,641
47
450
633

1

The actual size of `wchar_t` is implementation defined: It is 16 bits for Windows compilers and 32 bits for most UNIXes. It is at least 16 bits, though. I don't know what the relevant section is but 5.3.3 [expr.sizeof] paragraph 1 has a note stating this fact: "[ Note: in particular, sizeof(bool), sizeof(char16_t), sizeof(char32_t), and sizeof(wchar_t) are implementation-defined.74 —end note ]" – Dietmar Kühl Dec 02 '12 at 01:34
I have provided a 'sane' approach with a simple Spirit parser. – sehe Dec 02 '12 at 01:38
You know, I had a problem that was vector related earlier today and I got "yelled" at for using fread() instead of fstream to read into the vector. fread() was working fine, but I decided to try fstream for "C++ approach sake" as stated in the comments. But now I'm doubting fstream's practicality with binary files if it takes much more work to be able to use it for what seems to be wasting my time. – Tek Dec 02 '12 at 01:42
@Tek: There's nothing wrong with using fread. You can also use the fstream `read` function, if you find this usage of `std::copy` for reading binary files as distasteful as I do. – Benjamin Lindley Dec 02 '12 at 01:56
@BenjaminLindley Thanks, I think I will keep using at least the read function for now since it's less arguments. I'll keep the question open to see if any other interesting suggestions come up. – Tek Dec 02 '12 at 02:07
@Tek I'd say in some cases the `fread` or `istream::read` are perfect candidates. However, I'm a big proponent of _coding intent_ (***what*** do you want happening) over _coding steps_ (***how*** you think that should be achieved). In particular, Spirit will give you the means to just flip the switch and specify the endianness of the input, should you ever run into such a need. You know, that kind of stuff often becomes important in binary protocols – sehe Dec 02 '12 at 11:22

Getting desired binary data ranges from std::istreambuf_iterator and std::ifstream

2 Answers2

Linked