Boost loop performance when reshaping huge amount of data

Question

This is an extended question from previous question here. I'm reshaping the data according to the code below. However when the data size get increased, in=rand(291081,1920);, the answer (preallocation) in the previous question can't handle it anymore, MATLAB even freezed the whole laptop. Thus as suggested by Teddy in the last question, I'm opening a new question to boost the performance of the loop.

Some quite similar but not alike question can be found here. From there, the answer given suggested to modify the loop to be column-wise. However, since my loop required to access row and column simultaneously, it seem the answer is not applicable in my case.

Is this loop can still be modified to improve its performance? Or can this loop be done without freezes the whole laptop? If possible, without getting involve with GPUs.

in=rand(291081,1920);
m=581;
[R,C]=size(in); 
R_out=R/m; 

out=zeros(m*C,R_out);
for k=1:m %from row 1 to nth row
    for i=1:C %reshape every column of nth row
        out(i+C*(k-1),:) = in(k:m:end,i)';
    end
end

P/S: In previous question, when the data size getting bigger, the loop and arrayfun seem to not have large performance difference.

Thanks in advance!

score 3 · Answer 1 · answered Apr 02 '17 at 06:43

You could transpose you matrix and reshape it to only have R_outcolumns.

% Input
in=rand(291081,1920);
m=581;
[R,C]=size(in); 
R_out=R/m; 

% Reshape
out = reshape(in.', [], R_out);

The most of the time needed for this operation is caused by the transposing of the in matrix. So to speed it upp further you could perhaps try to collect/feed the data so that it's already transposed and then just do the reshape.

score 0 · Accepted Answer · edited May 23 '17 at 12:09

Firstly, thanks @NLindros for providing the answer, which had inspired this answer. It seems using reshape is so far the most efficient way to handle huge data. Thus to avoid the usage of transpose as suggested, permute and reshape is used, as below.

out=reshape(permute(reshape(in,m,R_out,[]),[1 3 2]),[],R_out);

This by far is the fastest way to reshape.

I'll mark this as answer if there's no more better answer.

Boost loop performance when reshaping huge amount of data

2 Answers2

Linked