How to quickly concatenate/stack lots of numpy arrays?

Asked May 07 '22 at 15:45

Active May 07 '22 at 16:03

Viewed 103 times

I have a really big matrix (3.600 x 30.000) which is a np array of np arrays, and I want to do the following process (this is not actual code!) to obtain the variance column-wise:

columnsArray = []
for column in matrix:
    newColumn = some_processing(column)
    columnsArray = np.append(columnsArray, newColumn)
variance = np.var(columnsArray)

I need it in order to calculate the variance of the processed values.

The problem is that the more columns I append, the slower the process becomes.

I also tried with np.column_stack, but it doesn't change much.

I'm sure there is a simpler way.

edited May 07 '22 at 16:03

asked May 07 '22 at 15:45

Alessandro

1

Use list append instead. It works in-place and just adds a reference to the list. Do you `concatenate` just once, after the loop. – hpaulj May 07 '22 at 16:09
See: https://stackoverflow.com/a/29840311/12939557 – Jérôme Richard May 07 '22 at 16:29

How to quickly concatenate/stack lots of numpy arrays?

0 Answers0