[New problem raised ]Duplicate elements inside nested list

Question

I have 3 lists

X1 = ['A','B','C','D']
X2 = ['A','1','2','3']
X3 = ['1','W','X','Y','Z']

I run through iterations to create all combinations of the lists

ALL = [X1,X2,X3]
a = list(itertools.product(*ALL))

Output:

[('A', 'A', '1'), ('A', 'A', 'W'), ('A', 'A', 'X'), ('A', 'A', 'Y'), ('A', 'A', 'Z'), ('A', '1', '1'), ('A', '1', 'W'), ('A', '1', 'X'), ('A', '1', 'Y').......

Apart from looping all the elements inside the list, can I have another method to erase all the duplicate elements inside the list? What I want should be only

('A', '1', 'W'), ('A', '1', 'X'), ('A', '1', 'Y')....... and so on

**New Problems raised **

What if the 3 lists changed to

X1 = ['A_1','B','C','D']
X2 = ['A_2','1_1','2','3']
X3 = ['1_1','W','X','Y','Z']

Because A_1, A_2 belongs to the same group and I still want to regard them as duplicate elements. So what should I do to remove all these duplicate elements like the above example?

https://stackoverflow.com/questions/19416786/remove-duplicate-tuples-from-a-list-if-they-are-exactly-the-same-including-order — sander, Mar 02 '21 at 08:49
@schwobaseggl Oh sorry ('A', '1', '1') is out too. I didn't notice that. Thanks! — YIPYIP, Mar 02 '21 at 08:54
@Sandertjuhh Thank you for the suggestions but I am not removing duplicate list (Actually there are no duplicate lists at all ). What I want is removing the particular list with duplicate elements. — YIPYIP, Mar 02 '21 at 08:57

Equinox · Answer 1 · 2021-03-02T11:35:07.417

2

You can use a list comprehension to remove the elements like so.

arr = itertools.product(*ALL)
ans = [ele for ele in arr if len(set(ele))==3]

Output:

[('A', '1', 'W'),
 ('A', '1', 'X'),
 ('A', '1', 'Y'),
 ('A', '1', 'Z'),
 ('A', '2', '1'),
  ...

edited Mar 02 '21 at 11:35

answered Mar 02 '21 at 08:58

Equinox

6,483
3
23
32

No need for the spurious in memory list. `arr = itertools.product(*ALL)` will do! – user2390182 Mar 02 '21 at 10:47
Yeah no need to convert into a list. Updated. – Equinox Mar 02 '21 at 11:34

score 1 · Answer 2 · answered Mar 02 '21 at 10:46

Instead of overproducing (a potentially vast amount of) combinations and filtering those down, you could also preprocess the base lists not to contain any more equal elements:

s = set()
for lst in ALL:
    new_lst = []
    for x in lst:
        if x not in s:
            s.add(x)
            new_lst.append(x)
    lst[:] = new_lst

a = list(itertools.product(*ALL))

Depending on your real data, this may reduce the run time dramatically.

[New problem raised ]Duplicate elements inside nested list

2 Answers2