How do I find two similar words within a list, and remove one of them?

Question

I have a list that looks something like,

inventory = ['Note', 'Key', 'Coin x5', 'Torch', 'Coin x8']

And I need the program to automatically remove the older duplicate string, in this case 'Coin x5', but I'm having trouble because the two strings aren't an exact match. If it helps, this list exclusively gets added to via the append function and does NOT get sorted alphabetically or any other way, so older items always have lower indexes than newer ones.

Is there always an index provided if an item has duplicates? If so, you could do a bit of string parsing to get the indices, analyse them and update the list accordingly. — FObersteiner, Jul 21 '19 at 13:53
What do you consider "similar"? Are there always just individual words separated by a space from their count? Why don't you use a ``dict`` or ``Counter`` to store the count for each item? — MisterMiyagi, Jul 21 '19 at 14:44

score 1 · Answer 1 · answered Jul 21 '19 at 23:38

What items are considered to be similar? If you can define an exact similarity score (between two strings), then it's possible using any string distance metric. Here's the code (uses difflib):


    import difflib

    inventory = ['Note', 'Key', 'Coin x5', 'Torch']
    threshold = 0.85 # Your value here

    def add_new_item(new_item):
        # Finding most similar item
        most_similar = max([(item, difflib.SequenceMatcher(None, item, new_item).ratio()) for item in inventory], key=lambda x: x[1])
        # If it's very similar, then remove
        if most_similar[1] > threshold: 
            inventory.remove(most_similar[0])
        inventory.append(new_item)

    print('Old list:', inventory)
    add_new_item('Bag')
    add_new_item('Coin x8')
    print('New list:', inventory)

This yelds:


    Old list: ['Note', 'Key', 'Coin x5', 'Torch']
    New list: ['Note', 'Key', 'Torch', 'Bag', 'Coin x8']

A good list of string similarity metrics you can find here.

srinu mss · Answer 2 · 2019-07-21T14:17:38.520

Try this script

inventory = ['Note', 'Key', 'Coin x5', 'Torch', 'Coin x8']
list_temp=[]
final=[]
matched_list={}
for i in inventory:
    lent=int(len(i)/2)
    list_temp.append(i[:lent])
for inv in inventory:
    for k in list_temp:
        if inv.find(k) == 0:
            matched_list[k] = inv
for fin in matched_list.values():
    final.append(fin)
print(final)

Updated script:

final=[]
matched_list={}
for i in inventory:
    lent=int(len(i)/2)
    matched_list[i[:lent]] = i
for fin in matched_list.values():
    final.append(fin)
print(final)```

How do I find two similar words within a list, and remove one of them?

2 Answers2