0

I want read file (any file, tiny and big) with stream (ifstream and ofstream).

I use follow function, this function is good for tiny and medium file

Struct StreamPacket
{
 long int startOffset;
 std::vector<char> data;
}
CONST int STREAM_BUFFER = 15000;
std::ifstream stream;

stream.open(path, std::ios::in | std::ios::binary | std::ios::ate);

if (!stream.is_open())
    return std::vector<StreamPacket>();


// create a vector to hold all the bytes in the file
std::vector<StreamPacket> wholePacket;
while (stream.is_open())
{
    StreamPacket fileStream;
    fileStream.startOffset = stream.tellg();
    // read the file
    std::vector<char> data(STREAM_BUFFER, 0);
    stream.read(&data[0], STREAM_BUFFER);
    fileStream.data = data;
    wholePacket.push_back(fileStream);
}

stream.close();

return wholePacket;

but I can't read big file (example 8 GB) with it, and I have error within while loop, Error is : 

Unhandled exception at 0x7703B782 in program.exe: Microsoft C++ exception: std::bad_alloc at memory location 0x004FEEDC.

what is wrong? Where is my problem?

and for write I use this function:

void SaveToFile(CString path, CString filename, std::vector<StreamPacket> fileStream)
{
std::ofstream outfile(path + filename, std::ios::out | std::ios::binary);

if (!outfile.is_open())
    return;

for (size_t i = 0; i < fileStream.size(); i++)
{
    outfile.write(&fileStream[i].data[0], fileStream[i].data.size());
}
int a = 10;

//outfile.write(&fileStream[0], fileStream.size());
outfile.close();
}

is Correct?

tank you for help me

Mr.DeveloperCplus
  • 234
  • 1
  • 10
  • 1
    How many iterations are there before the error occurs? – Jay Jan 16 '18 at 07:18
  • Maybe you don't have enough memory to store the whole file inside? – Holt Jan 16 '18 at 07:23
  • 2
    Is your application 32-bit? It looks like you're trying load the entire file into memory at once. The maximum memory a 32-bit application has is 4GB. The point of using a stream is to process the data as you read. – Disillusioned Jan 16 '18 at 07:23
  • @Jay very, exactly 135352 – Mr.DeveloperCplus Jan 16 '18 at 07:32
  • @CraigYoung tanks, but if I want read more of 4 GB in 32 application what is solution? – Mr.DeveloperCplus Jan 16 '18 at 07:38
  • 1
    It looks like you are running out of memory for your application. You can still read the file, but you will not be able to hold the entire contents in memory at one time. – Jay Jan 16 '18 at 07:43
  • @Mr.DeveloperCplus I guess you will have to move it to another type of memory once you are done with it (your hard drive or whatever you can use). So load into memory what you need, work with it, save it, load other chunk, work with it save it etc. Or you could move to 64 bit machine. – Blasco Jan 16 '18 at 07:46
  • The 4GB address space limit in a 32-bit OS is a hard limit (there are only 32 bits for addresses and 2^32 = 4GB). It's worse on Windows, the system reserves half of that address space for the OS so user applications only get 2GB. Look for a 64-bit toolset or rewrite your code to handle the data in smaller chunks. – Blastfurnace Jan 16 '18 at 08:00
  • 1
    @WooWapDaBug 64 bit machine doesn't help if app is 32-bit. You still have an upper limit of 4GB (practically only about 3.5GB). – Disillusioned Jan 16 '18 at 08:02
  • @WooWapDaBug yah, tank you, tanx, 1 question, above function in tiny file exactly not true, few byte is more than, what is wrong? in last of vector of byte, meybe half of vector fill in, but in write function that write entire vector – Mr.DeveloperCplus Jan 16 '18 at 08:05
  • @Mr.DeveloperCplus Don't try to read the entire file at once - ***it's not possible***. Read a bit, process a bit, read a bit, process a bit, etc. – Disillusioned Jan 16 '18 at 08:05
  • @CraigYoung ok, I got it, tanks, now, I have a small question but it is very Confusing.above function in tiny file exactly not true, few byte is more than, what is wrong? in last of vector of byte, meybe half of vector fill in, but in write function that write entire vector. how can i fix this? – Mr.DeveloperCplus Jan 16 '18 at 08:09

1 Answers1

0

Apart from 32 bits limit, the code has two big flaws.

a) you code is plentty of errors, so it takes time only to fix it.. (for example Struct StreamPacket must be "struct" in lowercase..

b) the logic behind files, since 1970 when RAM was very limited, is to read in chunks, process it, FREE/reuse buffer, and loop.

it allows to manages even TB of data using a very limited small footprint.

so the main idea is to re-think your code to use this logic. Can be more complex to write, but thinking managing file as in JS/Web app, is wrong.

a suggestion for using large file as memory: memory mapped files.

(for example at: mmap() vs. reading blocks)

or under windows similar functions exist.

ingconti
  • 10,876
  • 3
  • 61
  • 48