2

I need to create a very large array. Let us say 50 megabytes.

Can I be safe to create it as a normal static array? Will the compiler put it on the stack (possibly causing a stack overflow), or will it be smart enough to put it on the heap?

If there is no way to do so, is there an easy way to do it with malloc or "new" when program starts, but automatically free it when program ends?

trincot
  • 317,000
  • 35
  • 244
  • 286
Pubby
  • 51,882
  • 13
  • 139
  • 180

5 Answers5

3

As I understand it, static variables don't live on the stack. If they did, where would they go when you pop the stack frame they live in? Static function variables need to keep their state between calls, so logically, the static data should be kept on the heap.

Also, when the program ends, everything is automatically deallocated.

Twisol
  • 2,762
  • 1
  • 17
  • 17
  • Ah, I guess they won't be on the stack then. – Pubby Jan 31 '11 at 23:06
  • Everything is automatically deallocated on the stack, not on the heap. – Charles Ray Jan 31 '11 at 23:10
  • Aren't they both within the memory allocated for the process? Once the program ends, it's released back to the OS. – Twisol Jan 31 '11 at 23:12
  • 1
    @Twisol: They *memory* will be deallocated, but the *destructors* of the elements won't be called. – fredoverflow Jan 31 '11 at 23:16
  • @Twisol: memory management depends on the OS. The OS can swap memory segments to a "store" (swap file) depending on the needs for memory (on a system level). The C and C++ languages cannot guarantee that the *heap* is permanently in memory until the application exits. – Thomas Matthews Jan 31 '11 at 23:17
  • @Thomas Matthews: Excepting the under-the-hood details (and FredOverflow's point about destructors), my point was that there's no need to free memory if your program is about to quit anyways. Since the OP was asking about static variables, it seemed relevant. – Twisol Jan 31 '11 at 23:23
  • 1
    @Twisol: not running destructors is a major decision: it's very nice to have the ability to perform some statistics, logging etc. there, and some objects may use shared memory, temporary files or other resources that aren't automatically deallocated (on their OS). Further, habitually deleting memory makes it easier to spot actual memory leaks in valgrind, purify, insure etc.. – Tony Delroy Feb 01 '11 at 01:44
2

The easy way to do this is by using std::vector

std::vector   data;
data.reserve(<Number of Elements);

or potentially std::deque (depending on your usage).

Will the compiler put it on the stack (possibly causing a stack overflow), or will it be smart enough to put it on the heap?

A stack overflow is when the theoretical stack and theoretical heap collide and intermingle. If the stack was going to overfow then the heap is also going to fail.

Some systems have a maximum size of stackframe (this is compiler and platform specific) see your compiler documentation for details. As a result it is usually better to allocate huge structures dynamically (though not directly).

std::vector does this (probably). It has a small local object presence but the main payload (usually) is implemented as a dynamic heap allocation.

Martin York
  • 257,169
  • 86
  • 333
  • 562
  • I don't want to use std::vector. I guess I can take advantage of destructors and just wrap it in a class though. – Pubby Jan 31 '11 at 23:00
  • 3
    Because I asked about arrays. – Pubby Jan 31 '11 at 23:02
  • 3
    @Pepe: How is a std::vector different from an Array in the usage you want to perform? – Martin York Jan 31 '11 at 23:06
  • 1
    @Charles: What overhead does `std::vector` have that one's own class with a dynamically allocated array wouldn't have? Or even just a plain dynamically allocated array, for that matter? – fredoverflow Jan 31 '11 at 23:13
  • @Pepe: comment +1 because it cracked me up – John Dibling Jan 31 '11 at 23:21
  • std::vector will never be faster than correct use of arrays. It may achieve results as fast with full optimizations on, but it is always slow while debugging. Also, I find that I make stupid mistakes with vectors (like forgetting to reserve) that end up costing me more than just doing it with arrays. That is why I asked about arrays! – Pubby Jan 31 '11 at 23:23
  • @Pepe: Well, that's a refreshing perspective. – Twisol Jan 31 '11 at 23:25
  • @ Charles Ray: There is no overhead. Any decent compiler will make all normal usage the exact same cost as using an array (unless more space is needed and that can be compensated for by using reserve()). – Martin York Jan 31 '11 at 23:48
  • @Pepe: So your argument for not using arrays is that it is more error prone with vectors than with arrays? The argument about reserve is silly. If you forgot to reserve on a vector then you forgot to allocate on an array. The difference is the vector will compensate while the array is UB. – Martin York Jan 31 '11 at 23:50
  • vectors are almost always as fast after optimisation, and they're certainly safer and more convenient. The only slower operation I can think of: resize typically involves constructing a reference object and copying it to all other new elements - good for reference-counted objects, but unlikely to be optimised out when the constructor's a no-op. – Tony Delroy Feb 01 '11 at 01:38
  • @Martin: there's a certain logic to it... the UB's likely to be observed as a crash, while silent performance degredation's easily overlooked ;-). – Tony Delroy Feb 01 '11 at 01:39
  • @Tony: @ Pepe: This is worth a read to see effective speed differences: [stdvector is so much slower than plain arrays](http://stackoverflow.com/questions/3664272/stdvector-is-so-much-slower-than-plain-arrays/3664349#3664349) – Martin York Feb 01 '11 at 01:42
  • @Martin York: Proper use of vector gives you speed comparable to array (never faster). Improper use of vector gives you very bad performance. Proper use of array gives you fast speed. Improper use of array will likely crash program. Both require thought by programmer to implement in efficient forms, however I prefer to spend more time writing the code than running it. And besides, if you want to get full performance out of std::vector, you have to know how it works internally (and be constantly thinking about it), and so the abstraction really is pointless. – Pubby Feb 02 '11 at 20:15
  • "If the stack was going to overfow then the heap is also going to fail." is simply wrong most of the time. – Ben Voigt Apr 20 '15 at 01:50
2

It's better, in my experience, to allocate such a large array on the heap (thus via new) - I have seen a program core dump on a unix system after allocating 2 MB on the stack... If you want automatic deletion, you could use a smart pointer (say, boost::scoped_array). But, since you mention "deleting it automatically when the program ends", you actually don't have to do anything - the operating system will reclaim all of your process's memory when it terminates.

Anyway, you really should use std::vector instead of a raw array.

Andrea Bergia
  • 5,502
  • 1
  • 23
  • 38
1

50 megabytes is not too much by today standards.

You can allocate it at the start of your program using C++ new operator, and deallocate it with delete[] at end (or at the start/end of a defined program section).

If this array represents e.g some file to be loaded, it's better to allocate it of course when the file is loaded into memory. Optimally, you can only map only a section of the file into memory (e.g: 1MB, 2MB, or another logical "unit" you want to use) (see MapViewOfFile in Windows and mmap in UNIX systems). This way you can load very large files without exhausting your virtual memory.

Hernán
  • 4,527
  • 2
  • 32
  • 47
1

If you allocate it statically, it'll be allocated statically. In a typical case, there will be a record of some sort in the executable that specifies that a particular variable should be a zero-initialized block of size N. The loader will normally honor that, just like it allocates space for the program's code and such (e.g., it'll allocate address space but quite often not actual memory to back it until/unless you actually read/write that memory).

Jerry Coffin
  • 476,176
  • 80
  • 629
  • 1,111