1

Lets say I have a list as following:

points = np.array( [ . . . ] , [ . . .], [ . . .] )  # shape is (500000, 1000)

and another list as

target = np.array([ . . .])  #shape is (1000,)

Now I can compute the L2 norm as following:

norm = np.linalg.norm(points - target, axis=1)

This works perfectly but its super slow when I want execute this for 100K target values. For the moment I get a target value from the targets list and calculate norm for each target.

Is there a quick way to do this?

user1798933
  • 981
  • 1
  • 7
  • 7
  • 3
    500000 * 1000 is a lot of elements. 500k * 100k is even more. – Reti43 Jan 06 '16 at 09:24
  • For 64-bit numbers, 500k * 1k elements is already almost 4GB. With 500k * 100k elements, you'll probably run out of memory. Maybe you could try using something like [pytables](http://www.pytables.org/)? [Here's](http://stackoverflow.com/questions/14351255/techniques-for-working-with-large-numpy-arrays) another link that could help. – Praveen Jan 06 '16 at 11:34
  • Does the last sentence mean you are doing an iteration on top of the `norm` calculation? If so show that code. Otherwise answers will focus on memory size. – hpaulj Jan 06 '16 at 12:21

1 Answers1

3

Yep, broadcasting.

deltas = points[:, None, :] - targets[None, :, :]  # shape (n_points, n_targets, n_dims)
norm = np.linalg.norm(deltas, axis=2)  # shape (n_points, n_targets)

But you might use up all your memory if you have 500k points and 100k targets and 1000-D (that's 500000*100000*1000*8 B = 400 TB of RAM you need!)

Chances are your current slowdown is due to the actual cost of all that computation and not the fact that you're looping in Python. Consider using GPU computation (maybe Theano) to speed things up if you need.

Peter
  • 12,274
  • 9
  • 71
  • 86