1

What is a short, readable way to declare a 999x999 numpy matrix where each row is [1,2,3,...,999]? The final matrix should be:

[[1,2,3,...,999]
[1,2,3,...,999]
...
[1,2,3,...,999]]
jpp
  • 159,742
  • 34
  • 281
  • 339
riqitang
  • 3,241
  • 4
  • 37
  • 47

2 Answers2

5

You can use numpy.tile:

import numpy as np

res = np.tile(range(10), (5, 1))

print(res)

array([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
       [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
       [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
       [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
       [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]])

Alternatively, you can add to an array of zeros:

res = np.zeros((5, 10)) + range(10)
jpp
  • 159,742
  • 34
  • 281
  • 339
  • oh and also this syntax seems to work: `np.array([[x + 1 for x in range(999)] for y in range (999)])`, I was just confused by how the `print` method output the result so I thought I was wrong (should've checked the shape) – riqitang Jul 30 '18 at 21:55
  • 1
    @riqitang, The syntax works, but it's not recommended. List comprehensions are not vectorised. You will find the method inefficient versus NumPy methods. – jpp Jul 30 '18 at 21:55
  • 1
    It’s not just a matter of being not vectorized: you’re building a list of 999 lists of 999 elements, just to convert it to an array and throw it away. Building a big list like that takes time—and memory allocation. In non-numpy Python you usually avoid that by writing your code to use an iterators that only produces new rows, or even new cells, one at a time on demand. In numpy, you usually avoid it by creating an array directly with numpy operations. – abarnert Jul 30 '18 at 22:00
  • 1
    @abarnert, Yep, agreed. Vectorised is the wrong term. NumPy works by pre-allocating memory, while a list comprehension gradually increases memory allocation. [This is a good question](https://stackoverflow.com/questions/51526242/why-do-two-identical-lists-have-a-different-memory-footprint) giving insight into list comprehension memory allocation. Of course, *any* list which needs to be created before constructing a NumPy array will lead to inefficiency. – jpp Jul 30 '18 at 22:02
  • @jpp It's really not the gradual increasing that's the issue—the allocation increases geometrically, so the total allocation only ends up larger by a smallish constant factor. It's building the unnecessary list in the first place that matters—that's slower than not doing anything by a factor of `O(infinity)`. :) – abarnert Jul 30 '18 at 22:32
  • @abarnert, Indeed, that wasn't my point. But the question I linked to still demonstrates how memory allocation is *increased* sequentially in steps, rather than in one swoop. It's also just useful to know that a list comprehension does things slightly differently. – jpp Jul 30 '18 at 22:37
1

@jpp answer is elegant but the following solution is more efficient:

res = np.empty((nrows, ncols))
res[:, :] = np.arange(ncols)

Timing:

%timeit a = np.empty((1000,1000)); a[:, :] = np.arange(1000)
445 µs ± 9.08 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit np.tile(range(1000), (1000, 1))
1.43 ms ± 15.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Further timing tests:

Following @jpp comments I added one more test done in Python interpreter directly (unlike the original test that was run in a Jupyter notebook - because it was up and running at that moment):

>>> import sys
>>> print(sys.version)
3.6.5 |Anaconda, Inc.| (default, Apr 26 2018, 08:42:37) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
>>> import numpy as np
>>> print(np.__version__)
1.13.3
>>> import timeit
>>> t = timeit.repeat('res = np.empty((nrows, ncols)); res[:, :] = np.arange(ncols)', setup='import numpy as np; nrows=ncols=1000', number=100, repeat=50)
>>> print(min(t), max(t), np.mean(t), np.std(t))
0.04336756598786451 0.053294404002372175 0.0459639201409 0.00240180447219
>>> t = timeit.repeat('res = np.tile(range(ncols), (nrows, 1))', setup='import numpy as np; nrows=ncols=1000', number=100, repeat=50)
>>> print(min(t), max(t), np.mean(t), np.std(t))
0.05032560401014052 0.05859642301220447 0.0530669655403 0.00225117881195

The results with numpy 1.14.5 are virtually identical:

>>> import sys
>>> print(sys.version)
3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:07:29) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
>>> import numpy as np
>>> print(np.__version__)
1.14.5
>>> import timeit
>>> t = timeit.repeat('res = np.empty((nrows, ncols)); res[:, :] = np.arange(ncols)', setup='import numpy as np; nrows=ncols=1000', number=100, repeat=50)
>>> print(min(t), max(t), np.mean(t), np.std(t))
0.04360878499574028 0.05562149798788596 0.04657964294136036 0.0025253372244474614
>>> t = timeit.repeat('res = np.tile(range(ncols), (nrows, 1))', setup='import numpy as np; nrows=ncols=1000', number=100, repeat=50)
>>> print(min(t), max(t), np.mean(t), np.std(t))
0.05024543400213588 0.06169128899637144 0.05339125283906469 0.00276210097759817
AGN Gazer
  • 8,025
  • 2
  • 27
  • 45
  • 1
    I actually see `np.tile` twice as fast as the version initialised with `np.empty` (Python 3.6.5 / NumPy 1.14.3). Can you advise your versions? – jpp Jul 30 '18 at 22:07
  • 1
    I get 1.41ms for the `empty` version and 1.46ms for the `tile` version (NumPy 1.15.0, CPython 3.7.0, 64-bit, python.org macOS installer). Which just goes to show that anyone who cares about performance really needs to test on their actual system with their actual data, not just take what some guy on the internet says as universally true. – abarnert Jul 30 '18 at 22:35
  • For extra fun, try it with PyPy using native NumPy vs. NumPyPy; holding all else equal, they seem to give opposite results here. – abarnert Jul 30 '18 at 22:40
  • 1
    Weird that your Jupyter notebook took 3x as long to run the same code as the plain REPL for the second implementation, especially since it was virtually identical with the other implementation. Rather than try to guess at why, I'll just stick with "the OP had better test this himself if it matters." – abarnert Jul 30 '18 at 23:50
  • @abarnert I agree – AGN Gazer Jul 30 '18 at 23:55