"vector too long" error while filling std::vector

Question

i am trying to fill a two dimensional vector(817 by 577) with objects called cells. These cells have a set of member values (floats, other vectors etc). At a certain point the program stops and throws the error "vector<T> too long". Here is the class definition of cells and the complete loop:

struct cell
    {
    int x;
    int y;
    int country;
    vector<int> popO;
    vector<int> popS;
    vector<float> Rainfall;
    double Cropland;
    vector<movement> outm;
    vector<movement> inm;
    vector<double> AgeMaleFemale;
    vector<double> AgeMaleFemaleMortality;
    double Fertility;
    };
vector<vector<cell>> cells;
void fillCells(dataserver D)
        {
        cout<<"start filling"<<endl;
        int rows=577;
        int columns=817;
        cell X;
        vector<vector<cell>> output(rows,vector<cell>(columns,X));
        cout<<"start loop"<<endl;
        for (int i=0;i<rows;i++)
            {
            cout<<i<<" ";
            for (int j=0;j<columns;j++)
                {
                int p=-9999;
                cell tmpC;
                tmpC.x=i;
                tmpC.y=j;
                tmpC.country=D.CO[i][j];
                tmpC.popO.resize(3,0);
                tmpC.popO[0]=int(D.PO[0][i][j]);
                tmpC.popO[1]=int(D.PO[1][i][j]);
                tmpC.popO[2]=int(D.PO[2][i][j]);
                tmpC.Rainfall.resize(10,0);
                for (int k=0;k<10;k++)
                    {
                    tmpC.Rainfall[k]=D.R[k][i][j];
                    }
                tmpC.popS.resize(10,0);
                tmpC.Cropland=D.CPC[i][i];
                if (tmpC.country!=-9999)
                    {
                    tmpC.Fertility=D.F[tmpC.country];
                    tmpC.AgeMaleFemale.resize(18,0);
                    tmpC.AgeMaleFemale=D.AMF[tmpC.country];
                    tmpC.AgeMaleFemaleMortality.resize(18,0);
                    tmpC.AgeMaleFemaleMortality=D.M[tmpC.country];
                    }
                output[i][j]=tmpC;
                }
            }
        cells=output;
        }

Googling a bit I found out that sizeof(cell) times the number of cells in the vector is supposed to smaller than vector::max_size()

sizeof(cell) is 144 ->144*817*577=67882896

max_size is 268345455

Shouldn't there be enough space for all the cells then, or am I missing something? Thanks in advance!

Some additional Info:

Running on Windows 7 64bit, compiling with Visual Studio 2010, 32 bit

The info about max_size actually came from here: stl "vector<T> too long"

Why not print the vector's length once in a while to _verify_ your assumption of its length? — Lightness Races in Orbit, Dec 02 '13 at 15:23
*Googling a bit I found out that sizeof(cell) times the number of cells in the vector is supposed to smaller than vector::max_size()* I don't quite know where you got that from, but it is not *correct* (something similar might be true, but I think you misinterpreted whatever you read) — David Rodríguez - dribeas, Dec 02 '13 at 15:25
Please, take the time to copy the exact error message you are getting with as much context information (on the error) as possible — David Rodríguez - dribeas, Dec 02 '13 at 15:26
`max_size - This value is typically equal to std::numeric_limits::max(), and reflects the theoretical limit on the size of the container. At runtime, the size of the container may be limited to a value smaller than max_size() by the amount of RAM available.` — DumbCoder, Dec 02 '13 at 15:30
Which system are you using? Windows? Which compiler? std::vector is a data-structure with an array like semantic and the ability to dynamically increase its size at runtime. Since you don't seem to need the latter feature maybe you should just use a common 2D array? — Martin Meeser, Dec 02 '13 at 15:30
@DumbCoder - don't know where that quote came from, but it's wrong. `max_size()` is the maximum **`size`**, that is, the largest number of elements; it's typically `std::numeric_limits::max() / sizeof(element)`. — Pete Becker, Dec 02 '13 at 15:52
You have no vector here larger than 817 elements (specifically 577 of those). I'm fairly confident those numbers are significantly lower than any theoretical limits on a vector's indexable reach. Your problem is somewhere else. — WhozCraig, Dec 02 '13 at 15:55
@PeteBecker http://en.cppreference.com/w/cpp/container/vector/max_size — DumbCoder, Dec 02 '13 at 16:01
Pete Becker vs whoever wrote that line on cppreference.com in battle of "general statements about what most C++ implementations do" ;-) — Steve Jessop, Dec 02 '13 at 16:03
@SteveJessop - C++11 says that `max_size()` returns "`distance(begin(), end())` for the largest possible container. So, sure, it's possible that that statement reflects non-conforming implementations, but that's not useful information. Not that `max_size()` itself is useful, of course. — Pete Becker, Dec 02 '13 at 16:53
@PeteBecker: I think the ambiguity is in the word "possible" -- presumably if an implementation happens to know that its own memory allocator has an overhead of a few bytes, it's not required to report `max_size` as `(std::numeric_limits::max() - a_few) / sizeof(T)` for containers using `std::allocator`. — Steve Jessop, Dec 02 '13 at 16:57
@WhozCraig yes, thats what I thought, but where else could the problem be? I mean it definitely occurs within this loop — user2160180, Dec 02 '13 at 17:01
@SteveJessop - granted, the requirement is essentially meaningless, so pretty much any large number is valid, but good engineering still demands doing as good a job as possible, not just throwing out a value that will never be wrong. It's not the allocator, but the size of the address space and the size of the element that determines the maximum container size. — Pete Becker, Dec 02 '13 at 17:02
@LightnessRacesinOrbit isnt the vector length defined with rows and columns? I mean the size is not changing, only the sizes of small vectors within a cell — user2160180, Dec 02 '13 at 17:03
@PeteBecker: I guess there are trickier cases than `vector` anyway. For example the implementer knows for sure that a `std::list` requires more than one byte per element, so presumably should divide by the size of whatever node structure is used internally. — Steve Jessop, Dec 02 '13 at 17:06
So another (unrelated) question, was it your intention to pass your data server by-value ? (that seems to be a theme in this code, as there is a *ton* of copying going on, but I was just curious). — WhozCraig, Dec 02 '13 at 17:10
@WhozCraig no, not intentional, should I pass a pointer instead? (If so, could you briefly explain why?) — user2160180, Dec 02 '13 at 17:12
@user2160180 - sorry for the digression about `max_size()`. To find out what's going wrong, start commenting out code until the problem goes away. So, for example, don't store anything into `tmpC`; just stuff the default-initialized cell into `output`. If that works, start adding back bits and pieces of the data until you've identified the culprit. — Pete Becker, Dec 02 '13 at 17:18
@user2160180: My point is that you have a problem that you don't understand, so to debug it you need to _verify_ your assumptions. You'll find that at least one of them doesn't hold, else you wouldn't be here! — Lightness Races in Orbit, Dec 02 '13 at 17:19
@user2160180 regarding all the copying, you can probably substantially reduce it. [possible example](http://pastebin.com/Me3d6mEW) — WhozCraig, Dec 02 '13 at 17:24

score 10 · Accepted Answer · answered Dec 02 '13 at 16:05

10

Since I havent got enough rep to comment, so you get it as an answer.

The error text sounds like you are on a windows machine http://msdn.microsoft.com/en-us/library/2af6btx2%28v=vs.80%29.ASPX

And I think it has to do with a resize, since creating a vector that is too big throws bad_alloc (on linux, so it might be different on windows).

Try and wrap all resize calls in a

try {
    // resize call here
} catch (Exception &e) {
    std::cerr << e.what() << std::endl
              << "Some text that identifies the line" << std::endl;
}

answered Dec 02 '13 at 16:05

Silas

468
3
7

6

This is why rep barriers don't work. People assume the rep barriers are there to force them to break the system, rather than say "oh ok, I'll shut up for now then". Annoyingly, this is a poor example because your "comment" is insightful and useful. Thanks for posting it! I mean, really, this _is_ an answer... – Lightness Races in Orbit Dec 02 '13 at 17:17
1

While this did not provide the answer, wrapping different block in try statements to see exactly which call causes the exception did help. Its "tmpC.Fertility=D.F[tmpC.country];" It seems to bee an index out of bounds problem, since the country index begins with 1, the vector with 0. So thanks a lot! – user2160180 Dec 02 '13 at 17:23

"vector too long" error while filling std::vector

1 Answers1