How to remove duplicated element in list in python3?

Question

environment: python 3.6.4

I have two list,
list1 is nested list of words, like

[['this', 'is', 'a', 'pen', 'that', 'is', 'a', 'desk'],
 ['this', 'is', 'an', 'apple']]

list2 is list of words to remove from list1 , like

['a', 'an']

I want to get new list like

[['this', 'is', 'pen', 'that', 'is', 'desk'],
 ['this', 'is', 'apple']]

and won't change list1.

I wrote below code, but my code destroy list1, where's wrong my code?

def remove_duplicate_element_in_nested_list(li1, li2):
    """
    :param li1: <list> nested_sentences
    :param li2: <list> words_to_remove
    :return: <list>
    """
    ret = []
    for el1 in li1:
        ret.append(el1)

    for i in range(len(ret)):
        for el2 in li2:
            try:
                # list.remove() remove only one element. so loop this.
                for el in ret[i]:
                    ret[i].remove(el2)
            except ValueError:
                None

    return ret

words = [['this', 'is', 'a', 'pen', 'this', 'is', 'a', 'desk'], ['this', 'is', 'an', 'apple']]
stop_words = ['a', 'an']

print(words)
# shows [['this', 'is', 'a', 'pen', 'that', 'is', 'a', 'desk'], ['this', 'is', 'an', 'apple']]
new_words = remove_duplicate_element_in_nested_list(words, stop_words)
print(words)
# shows [['this', 'is', 'pen', 'that', 'is', 'desk'], ['this', 'is', 'apple']]

Answer 1

ret.append(el1) will not copy the inner list, it copies the reference to the inner list instead.

Try using ret.append(el1[:]) which uses the slice operator to create a copy. Other methods of creating a copy of a list are illustrated here: How to clone or copy a list?

Answer 2

A simple for loop method.

def remove_duplicate_element_in_nested_list(li1, li2):
    """
    :param li1: <list> nested_sentences
    :param li2: <list> words_to_remove
    :return: <list>
    """    
    ret = []
    for i in li1:
        r = []
        for k in i:
            if k not in li2:
                r.append(k)
        ret.append(r)

    return ret

A = [['this', 'is', 'a', 'pen', 'that', 'is', 'a', 'desk'], ['this', 'is', 'an', 'apple']]
B =  ['a', 'an'] 
print(remove_duplicate_element_in_nested_list(A, B))

Result:

[['this', 'is', 'pen', 'that', 'is', 'desk'], ['this', 'is', 'apple']]

Answer 3

The problem in your code is with this line

ret.append(el1)

Basically now li1 and ret, both contain the same inner lists. So when you do ret[i].remove(el2), it removes that from both li1 and ret.

You can get your code working by changing the line ret.append(el1) to ret.append(list(el1))

Answer 4

Because everything is object in python, and list is mutable. It is easy to test:

>>> lst = [[1], [2]]
>>> new_lst = []
>>> for e in lst:
...     new_lst.append(e)
...
>>> new_lst[0] is lst[0]
True
>>> new_lst[0].append(10)
>>> new_lst
[[1, 10], [2]]
>>> lst
[[1, 10], [2]]

copy.deepcopy is an advice

Answer 5

You must recognize that lists are mutable, and when you pass them to functions, they are references to the same object and can produce unexpected results if you're not aware of how that works. For example...

# BAD:

def filter_foo(some_list):
    while 'foo' in some_list:
        some_list.remove('foo')
    return some_list

This will alter the list passed to it as well as return the same list to the caller.

>>> a = ['foo', 'bar', 'baz']
>>> b = filter_foo(a)
>>> a # was modified; BAD
['bar', 'baz']
>>> b is a # they're actually the same object
True

The following avoids this problem by creating a new list

# GOOD:

def filter_foo(some_list):
    new_list = []
    for item in some_list:
        if item != 'foo':
            new_list.append(item)
    return new_list

The list passed was not modified and a separate list with the expected result is returned to the caller.

>>> b = filter_foo(a)
>>> a # not modified
['foo', 'bar', 'baz']
>>> b
['bar', 'baz']
>>> a is b
False

Though, that took a refactor. To fix places where you do this, a simple solution is to make a copy.

# Drop-in fix for bad example:

def filter_foo(some_list):
    some_list = some_list[:] # make a copy
    # rest of code as it was
    return some_list

A different, easy-to-read solution to your problem with simple recursion. Added some comments in case anything wasn't clear.

def filter_words(word_list, filtered_words):
    new_list = []
    for item in word_list:
        if isinstance(item, list):
            # if it's a list... filter that list then append it
            new_list.append(filter_words(item, filtered_words))
        # otherwise it must be a word...
        elif item in filtered_words:
            # if it's in our excluded words, skip it
            continue
        else:
            # it's a word, it's not excluded, so we append it.
            new_list.append(item)

Testing

>>> filter_words(l, ['a', 'an'])
[['this', 'is', 'pen', 'that', 'is', 'desk'], ['this', 'is', 'apple']]    

This should work no matter how deeply nested (up to the recursion limit) the lists are. Could be refactored to any desired level of nestedness, too.

Answer 6

My way to copying list is not copying values but copying reference.

 ret = []
 for el1 in li1:
     ret.append(el1)

In this case, I have to copy value, and the way are below.

ret.append(el1[:])

or

import copy
ret = copy.deepcopy(li1)

or

ret.append(list(el1))

or something else.

thanks a lot of answers.

Answer 7

Try this code

list1=[['this', 'is', 'a', 'pen', 'that', 'is', 'a', 'desk'],['this', 'is', 'an', 'apple']]
list2=['a', 'an']
for out in range(0, len(list1)):
  for _in in range(0,len(list1[out])):
    if list1[out][_in]==list2[out]:
       list1.remove(list1[0][1]);