Python - Compare list element if exist

Question

I have a large table couple of millions of pairs of ints [[1,2],[45,101],[22,222] etc..]. What is the quickest way in Python to remove duplicates ?

Creating empty list and appending it "if not in" doesn't work since it takes ages. Converting to Numpy and use "isin" I can't seem to get it to work on pairs.

I see you have asked a few questions over the years, but as a refresher, please remember that this is **not a discussion forum**. We [do not want conversational language](https://meta.stackexchange.com/questions/2950) here. As for the question: is it required to use lists for the pairs? This question is trivial if they can be converted to tuples. `numpy.isin` won't help much, because the problem is the algorithm, not Python overhead. — Karl Knechtel, Jun 03 '22 at 19:24
Please first read https://stackoverflow.com/questions/7961363/removing-duplicates-in-lists. Note that to use the `set` approach, your elements don't have to be ints, but they do have to be *hashable* - sub-lists won't work (which is why I ask about converting to tuple first). If this is enough information to solve the problem, I can mark the question immediately as a duplicate; otherwise, please clarify what else you need to know. — Karl Knechtel, Jun 03 '22 at 19:27

score 2 · Accepted Answer · edited Jun 03 '22 at 19:28

2

you can do the following

arr = [[1,2],[45,101],[22,222], [1,2]]

arr = set(tuple(i) for i in arr)

if you want to convert it back to list

arr = [list(i) for i in arr]

edited Jun 03 '22 at 19:28

Karl Knechtel

62,466
11
102
153

answered Jun 03 '22 at 19:27

demetere._

268
2
10

@Demeter - brilliant. Thank you for taking the time and guiding me here. Appreciate it ! – Error Replicator Jun 03 '22 at 19:57

thebadgateway · Answer 2 · 2022-06-03T19:31:05.283

0

Probably going to be this: list(set(my_list))

Edit: Whoops. In any case, if whatever is iterating over said list can perform the task of detecting duplicates, that’d be the faster than removing duplicates beforehand.

edited Jun 03 '22 at 19:31

answered Jun 03 '22 at 19:22

thebadgateway

433
1
4
7

This will throw an error "unhashable type: list" because the OPs list is a list of lists. You'd have to first convert the list to a list of tuples. Afterwards you might want to convert the tuples back to lists. – Luatic Jun 03 '22 at 19:25

score 0 · Answer 3 · edited Jun 03 '22 at 19:29

0

You could use np.unique():

np.unique([[1,2],[45,101],[22,222],[22,222]], axis=0)

Output:

array([[  1,   2],
       [ 22, 222],
       [ 45, 101]])

Note that this re-orders the list

edited Jun 03 '22 at 19:29

Karl Knechtel

62,466
11
102
153

answered Jun 03 '22 at 19:26

Nin17

2,821
2
4
14

Python - Compare list element if exist

3 Answers3