2

For the program I'm working on, I frequently need to read input from a text file which contains hundreds of thousands of integers. For the time being, I'm reading a handful of values and storing them in a vector. Whenever a value I need is not in the vector, I read from the input file again and flush out the old values to make room for the values I'm currently reading in.

I'd like to avoid a situation where I constantly need to read from the input file and I'm wondering how many values I can store in my vector before there will be a problem. max_size() returns 1073741823, so I'm thinking that I can store that many elements but I'm wondering where that memory is being used and if it's a good idea to have a vector that large.

When you create a vector as so:

int main(){
std::vector<int> vec;
vec.push_back(3);
vec.push_back(4);

return 0;
}

Is that vector now using stack memory? Since your vector contains 2 ints, does that mean that 8 bytes of stack memory is being used?

According to MSDN docs:

For x86 and x64 machines, the default stack size is 1 MB.

That does not seem like a lot of memory. What is an example of a situation where you would want to increase the stack memory? Is there any way in Visual Studio to monitor exactly how much stack and heap memory are currently being used?

Is there anything I can do to prevent constant reading from the input file in a situation like this?

user974967
  • 2,928
  • 10
  • 28
  • 45
  • How big are the files you are working with? If reasonable sized at all, I'd just load the whole thing into RAM and work from there. – Michael Dorgan May 29 '12 at 17:31
  • The input file is still being modified/added to, but in the end I'd except it to have maybe 350,000 integers. By reading the whole thing into RAM at once, do you mean dynamically allocating a buffer and then calling ifstream's get? – user974967 May 29 '12 at 17:48
  • 1Meg file, I'd probably just read the whole file into a single buffer and do my work off it. If the file is being changed on the fly by many other users, then an OS specific solution such as memory mapped file reads may be better. Performance-wise, a single 1 Meg read is going to be much faster than a ton of 4 byte random reads. – Michael Dorgan May 29 '12 at 17:55

2 Answers2

4

Is that vector now using stack memory?

The vec object is on the stack, but it internally allocates its memory on the heap as it grows

EDIT

Also, instead of reading all the file and storing it in a vector, you could try using a memory mapped file. From what I understand (not having used them myself), you would benefit from page caching and file reading in kernel mode (as the OS will manage the loading of the file on demand).

Note that this is merely a suggestion on where to pursue your investigation (I think that it might be appropriate, but I am not familiar enough with memory mapped files to tell you more)

J_D
  • 3,526
  • 20
  • 31
0

vector stores elements in the heap, not the stack. Whether you should really allocate that much heap memory is a different matter, but you won't blow your stack.

SirPentor
  • 1,994
  • 14
  • 24