2

Assume a simple example where I have indices 

index_pos = [3,4,5];
index_neg = [1,2];

I would like to have a matrix:

result =

     1     3
     2     3
     1     4
     2     4
     1     5
     2     5

For this purpose I write the following code:

[X,Y] = meshgrid(index_pos,index_neg);
result = [Y(:) X(:)];

I think this is not a very efficient way. Also, this uses too much of my memory when I use big instances. I get the following error:

Error using repmat
Out of memory. Type "help memory" for your options.

Error in meshgrid (line 58)
        xx = repmat(xrow,size(ycol));

Error in FME_funct (line 36)
[X,Y] = meshgrid(index_pos,index_neg);

Is there any 'clever' way to generate this matrix using less memory?

PS: I noticed that what I do is also given here. Most probably I have found this idea from there.

Cris Luengo
  • 55,762
  • 10
  • 62
  • 120
independentvariable
  • 318
  • 8
  • 18
  • comment: maybe this is a special vector product in linear algebra.. – independentvariable Jun 24 '19 at 15:19
  • It is not efficient and uses too much memory because `result = [Y(:) X(:)];` requires copying the data to a new array? You can avoid the copy if you implement this as a loop, would be pretty trivial. – Cris Luengo Jun 24 '19 at 15:34
  • 2
    Oh, the out-of-memory error occurs in `meshgrid`. That means you don't have enough memory for `X` and `Y`, which means you don't have enough memory to store the matrix you want to build. There's no way around that except buy more memory for your computer. You'll have to rethink your algorithm so that you don't need this very large array. – Cris Luengo Jun 24 '19 at 15:35
  • @CrisLuengo I think you are right. I copied the size and tried to initiate a zero matrix in that size. Even that's not possible. so I think that's due to my RAM :\ – independentvariable Jun 24 '19 at 15:48
  • Do you need the entire matrix all at once, or do you use single rows, or groups of rows, for processing and then move on to the next? – beaker Jun 25 '19 at 15:58
  • @beaker all :\\ – independentvariable Jun 25 '19 at 20:55

2 Answers2

4

This depends entirely on how big your two variables are in relation to the amount of memory in your computer (plus the types of numbers you're using). Try this:

res = zeros(numel(index_neg)*numel(index_pos), 2)

If that gives you an out-of-memory error then you don't have enough memory in your computer to store the result, regardless of the efficiency of the generator, so if the above errors, then you're stuck. If it does not error, then you could well write a looping algorithm that uses less temporary memory.

That said, by default MATLAB represents numbers with double precision, 8 bytes per number. If your index_ variables happen to contain, say, only positive integers (all less than 65,536) then you could use 16-bit unsigned integers. These are just 2 bytes per number and so take up 4 times less space than doubles. You can test this with:

res = zeros(numel(index_neg)*numel(index_pos), 2, 'uint16')

Finally you can find out how much memory is available to MATLAB with the memory command.

Justin
  • 1,980
  • 1
  • 16
  • 25
  • That's a great answer. I definitely know that the vectors are indexes of an array. So they are non-negative integers. Can I replace this in the meshgrid function ? – independentvariable Jun 24 '19 at 16:08
  • 1
    @independentvariable: You don't need `meshgrid`, you can use any of the other solutions in the Q&A you linked. For example [this one](https://stackoverflow.com/a/7447119/7328782) would be simple to do with `uint16` or `uint32` inputs. – Cris Luengo Jun 24 '19 at 16:34
  • You can convert your double indices using the `uint16()` or `uint32()`, etc functions. – Justin Jun 24 '19 at 18:02
0

Here is a faster way to generate such a matrix. It avoids explicit temporary arrays by building the matrix directly in place,

res2 = [ reshape( bsxfun( @times , index_neg.' , ones(size(index_pos)) ) , [] , 1 ) , ...
         reshape( bsxfun( @times , index_pos , ones(size(index_neg)).' ) , [] , 1 ) ] ;

Note that this require the same amount of memory to hold the main array, so it will not be possible to generate arrays larger than with your method (which fails at the meshgrid stage). This maximum size is ultimately dictated by the amount of RAM available to your system.

Hoki
  • 11,637
  • 1
  • 24
  • 43
  • I do not think you are creating a matrix in place here. MATLAB evaluates the two `bsxfun` functions separately, producing two intermediate matrices, which are then concatenated. The end result is identical to the code in the OP, requiring the same amount of intermediate memory. – Cris Luengo Jun 24 '19 at 16:29
  • @CrisLuengo, yep I know, I should have highlighted more the word _explicit_. It is just a way to delegate to MATLAB the management of temporary arrays ... (hoping they have some trick applicable in some cases which can save ressources). – Hoki Jun 24 '19 at 17:36