It's my understanding that you want to extract out rows of a matrix where they all share common values along the first and third columns. You can certainly use the arrayfun approach, but you'll need to modify how you call unique. Specifically, use the 'rows' flag on the columns that you want to examine, then inside your arrayfun call, use bsxfun combined with allto check for those rows that contain each unique combination of column elements you want, so:
>> M=[ 1 3 5 6; 3 6 5 4; 1 8 5 1; 4 6 5 7; 3 6 4 5; 3 6 4 4];
>> r = unique(M(:,[1 3]), 'rows', 'stable');
>> A = arrayfun(@(x) M(all(bsxfun(@eq, M(:,[1 3]), r(x,:)),2), :), 1:size(r,1),'uni', 0);
>> celldisp(A);
A{1} =
1 3 5 6
1 8 5 1
A{2} =
3 6 5 4
A{3} =
4 6 5 7
A{4} =
3 6 4 5
3 6 4 4
It's important to note that I use the 'stable' flag, because unique by default sorts the unique entries. 'stable' ensures that we find unique entries in the order in which we encounter them, like the desired output that is seen in your question. BTW, 'uni' is short for 'UniformOutput'. It'll save you typing in the long run :).
The third line is quite a mouthful, but very easy to explain. First, take a look at this statement:
bsxfun(@eq, M(:,[1 3]), r(x,:))
What we will do here is we take each row of r, which is a unique pairing of values taken from columns 1 and 3 of M, and see whether each value in a row of M is equal to the corresponding location in a row from r. In order to find a match, we need to make sure that all values from a row of this result are equal to 1, which is why we need to use all here and look across the columns:
all(bsxfun(@eq, M(:,[1 3]), r(x,:)),2)
Once we find these rows that match with a particular row in r, we use these and index into our matrix M and extract the rows that satisfy the row in r we are looking for in M and therefore:
M(all(bsxfun(@eq, M(:,[1 3]), r(x,:)),2), :)
x iterates from 1 up to as many rows as there are in r, and at each iteration, we extract out a unique row from r each time. The end result will be a cell array that groups rows of M based on common elements between the first and third columns.
If you want this to be more efficient, you can do what Divakar suggested and use only the third output of unique. This'll make things easier to read too:
>> M=[ 1 3 5 6; 3 6 5 4; 1 8 5 1; 4 6 5 7; 3 6 4 5; 3 6 4 4];
>> [~,~,r] = unique(M(:,[1 3]), 'rows', 'stable');
>> A = arrayfun(@(x) M(r == x, :), 1:max(r),'uni', 0);
>> celldisp(A);
A{1} =
1 3 5 6
1 8 5 1
A{2} =
3 6 5 4
A{3} =
4 6 5 7
A{4} =
3 6 4 5
3 6 4 4
Certainly more readable! What's happening now is that the third output of unique assigns a unique ID from 1 up to as many unique values that you have that serves as the input into unique. Then, we simply find the maximum ID, and iterate from 1 up to the maximum ID, where at each iteration, this index serves as a way of extracting out the rows that correspond to each unique combination of the first and third columns from the matrix M.
