Efficient way of reading part of a file into an std::vector?

Question

Refering to this question: Efficient way of reading a file into an std::vector<char>? I need a function that does the followig thing:

void readFromFile( std::vector< unsigned char >& buffer,
                   string filename,
                   size_t offset, size_t count );

so the function read from the file from offset to offset + count into vector;

void readFromFile( std::vector< unsigned char >& buffer,
                   string filename,
                   size_t offset, size_t count )
{
    // get file size and reallocate the buffer
    size_t fsize = filesize( filename );
    buffer.reserve( buffer.size() + size );

    // open the file
    ifstream file( filename );

    // first way
    file.seekg( offset );
    file.read( ???? )

    // second way
    istreambuf_iterator< unsigned char > from( file );
    istreambuf_iterator< unsigned char > eof;

    advance( from, offset );
    copy( from, eof, back_inserter( buffer );
}

In the first way I don't know how to read the file at once. In the second way the read operation is quite slow because I read byte per byte.

Are better alternatives?

EDIT

Thanks to @Ben Voigt

I wrote this two simple functions:

inline std::streamsize filesize( const std::string& filename )
{
    std::ifstream in( filename, std::ifstream::in | std::ifstream::binary );
    if ( !in ) throw std::invalid_argument
    {
        "filesize error: invalid filename"
    };

    in.seekg( 0, std::ifstream::end );
    return in.tellg();

    // here the file is closed. so no need to restore the get pointer
}

inline std::streamsize filesize( std::ifstream& file )
{
    file.seekg( 0, std::ifstream::end );
    const auto size = file.tellg();
    file.seekg( 0 );      // restore the get pointer
    return size;
}

template< typename RAIter >
inline void read_file( std::istream& file,
                       RAIter first, RAIter last,
                       std::streamsize offset = 0
                        )
{
    const auto size = last - first;
    file.seekg( offset, std::ifstream::beg );
    file.read( reinterpret_cast< char* >( &*first ), size );
}

template<>
inline void read_file( std::istream& file,
                       unsigned char*  first, unsigned char*  last,
                       std::streamsize offset /*= 0 no default argument in template spacalization. */
        )
{
    const auto size = last - first;
    file.seekg( offset, std::ifstream::beg );
    file.read( reinterpret_cast< char* >( first ), size );
}

so the function now became easy:

vector< unsigned char > buffer;
// do something with buffer

const string filename{ "blabla" };

const auto size = filesize( filename );

// resize the buffer
auto const OLD_LEN = buffer.size();
buffer.resize( OLD_LEN + size );

size_t startOffset = 0;       // from where to star reading from file
size_t cont = size;           // how manny bytes read from file

// read filename from startOffset to startOffset + count, appendeing in buffer
ifstream file( filename );
read_file( file,
           buffer.data() + OLD_LEN,
           buffer.data() + OLD_LEN + count,
           startOffset
           );

You can use `reserve` along with a `back_inserter` and `std::copy`, or `resize` which gets you a valid memory region to directly fill. — Ben Voigt, Aug 28 '13 at 18:52
Why don't you just `mmap()` the data into your virtual address space? Why do you need a `vector`? — cmaster - reinstate monica, Aug 28 '13 at 18:53
@cdhowie I'm always doing that. Maybe electro-shock therapy will finally break me of that habit. — WhozCraig, Aug 28 '13 at 18:54
Pity, but isn't there something equivalent to `malloc()` on non-POSIX-conformant systems like windows? — cmaster - reinstate monica, Aug 28 '13 at 19:16

Ben Voigt · Accepted Answer · 2013-08-28T19:00:26.837

3

auto old_end = buffer.size();
buffer.resize( old_end + blocksize );

//...

file.read( &buffer[old_end], blocksize );
auto actual_size = file.gcount;
if (actual_size < blocksize) buffer.resize(old_end + actual_size);

edited Aug 28 '13 at 19:00

answered Aug 28 '13 at 18:53

Ben Voigt

277,958
43
419
720

+1 I totally missed he wanted to *append* the new block to the existing buffer. (and I'm not entirely sure he does, but it would certainly make sense based on the sizing he's doing in the permeable). – WhozCraig Aug 28 '13 at 18:55
@WhozCraig: It wasn't all that clear, but he did expand the size in `reserve` instead of replacing it. – Ben Voigt Aug 28 '13 at 18:56
Nice touch for shrinking on a short-read too (was about to suggest that). – WhozCraig Aug 28 '13 at 18:58

score 0 · Answer 2 · answered Aug 28 '13 at 18:48

0

Here's a quick and efficient way of getting a character array from a file.

char * arr;
int len;
// Function that opens a file, needing the file name
void openFile(const char* fileName)
{
ifstream file(fileName, ios::in);

if(!file.is_open()) return;

file.seekg(0, file.end);
    // Get the length of the file
len = file.tellg();
file.seekg(0, file.beg);

arr = new char[len];
file.read(arr, len);

file.close();
}

After that you can just push the char array into a vector.

answered Aug 28 '13 at 18:48

Radnyx

210
2
14

2

`char arr[];` isn't a valid definition, and then you'll have a type mismatch at `arr = new char[len];`, and then you'll leak. Were you trying to write C#/Java there? – Ben Voigt Aug 28 '13 at 18:51
arr = new char[len]; is a gcc extension, not supported with visual c++ – Neil Sep 09 '13 at 13:56

Efficient way of reading part of a file into an std::vector?

2 Answers2