How to find intersection of two huge numpy arrays

Asked Dec 22 '18 at 14:10

Active Dec 22 '18 at 14:10

Viewed 40 times

I have two huge 1D np arrays of row sizes greater than 200,000

say a size ~200,000 and b size ~300,000

i want to find the values of a which are present in b.

so far i have done the below method, but it sure is bad since it is taking too long

elements_common = set([i for i in numpy.reshape(b.shape[0],)]).intersection(set([i for i in numpy.reshape(a.shape[0],)]))

there is too much going on.. i am reshaping, creating a list, creating a set and then finding the intersection. leading to huge time complexities.

are there any better ways. appreciate all your help

asked Dec 22 '18 at 14:10

user1906450

Why do you need to `reshape` if they are 1D arrays? I'm not following. – roganjosh Dec 22 '18 at 14:12
1

Do you need `np.intersect1d(a,b)` ? – jezrael Dec 22 '18 at 14:13
FYI anything < 1M isn't _that_ huge. – cs95 Dec 22 '18 at 14:14
hi jezreal, i will try your solution and get back to you – user1906450 Dec 22 '18 at 14:31
hi coldspeed, then why does python take so long to process something thats 200k size. – user1906450 Dec 22 '18 at 14:31

0 Answers0