-1

So I have a set of data which is similar to this:

index |Parameter A| Parameter B

1|1|3

2|1|5

3|1|1

4|2|12

5|2|15

6|2|41

7|3|22

8|3|14

9|3|9

I need to calculate all the possible combinations of parameters B with a different parameter A i.e. (1)3-(2)12-(3)9 and obtain the indices of said combinations, in this case 1-4-9. The way I solved this was defining the index at which parameter A changes (in this case 3 and 6, I should point out it is not periodic) and use the combvec function in a loop:

  • combvec(combvec(1:3,4:6),6:9);

An then obtain a matrix which has all the combination indices in each column, which I can then use to obtain combinations of parameter B. The thing is the list is too big now hence the Out of memory problem.

The objective here is to calculate for each sequence of parameters B obtain a certain result - X - and then choose the combination that minimises said result.

I thought I could divide the list in two but the problem is since X is dependent on all A parameters the minimum of both groups will not necessarily be the global minimum, so I am not so sure.

The other alternative I thought was try obtain each combination in a loop and immediately calculate X and only store X if it was smaller than the previous iteration. This solution would solve my problem of storage but I am not so sure how to do this without a lot of nested loops because I would not be able to use combvec.

I would appreciate if you guys had any ideas on how to solve this or how to come about my suggestions avoiding the problems I mentioned.

My thanks in advance.

Community
  • 1
  • 1
jcarvalho
  • 115
  • 2
  • 11
  • There are a couple of alternatives [here](http://stackoverflow.com/questions/21895335/generate-a-matrix-containing-all-combinations-of-elements-taken-from-n-vectors). How many combinations do you need to generate? – beaker Jan 08 '17 at 03:39
  • 3e9 possible combinations which would mean about 24GB, I guess. – jcarvalho Jan 09 '17 at 13:31
  • 1
    In general, that many computations is a big warning sign that you might need to rethink your approach. If you *really* **really** ***really*** have to do it, one cheap way is to take your largest vector and process each element one at a time like `combvec(1, b, c); combvec(2, b, c); combvec(3, b, c);` and so on. This will only divide the number of combinations processed at one time by the size of the largest vector, but maybe that's enough. You can generate combinations iteratively, but it is ***slow***. – beaker Jan 09 '17 at 14:58

1 Answers1

0

I posted this question on reddit and got and answer to it so I am going to copy paste the answer. The reasoning is quite similar to what the user beaker suggested:

A = 1:4;
B = 5:8;
C = 9:12;
D = 13:16;

AB = combvec(A,B); % 2x16
CD = combvec(C,D); % 2x16
ABCD = combvec(AB,CD); % 4x256
ABCD = combvec(combvec(combvec(A,B),C),D); % 4x256, same as above

ABCD nested combination gets big REALLY fast. Instead, we can loop.

looped = [];
for idx=1:length(CD)
   looped = [looped [AB; repmat(CD(:,1),1,length(AB))]];
end

For each possibility in CD, combine it with all of the AB possibilities. "looped" is the same result as ABCD. Rather than store each combination in the loop, we could test for minima This approach only needed 64 values in memory instead of 1024.

Credit to /u/BCPull

Original Post link: https://www.reddit.com/r/matlab/comments/5mo48r/out_of_memory_problem_with_combvec_function_for_a/

Unfortunately as the user beaker suggested I need to rethink my approach since exchanging storage for processing time uses a lot of computing time and its not a good idea in general.

jcarvalho
  • 115
  • 2
  • 11