Performance optimization when using two for loops for list comparison

Asked Sep 18 '20 at 06:07

Active Sep 18 '20 at 06:07

Viewed 48 times

I am currently trying to compare each element of one list with every element of another list. This has a time complexity of N*N which is a hindrance to the performance.

The loops are as follows:

#fin_textt is a list which contains strings

for i in (range(len(fin_textt))):
    for j in range(i+1,len(fin_textt)):
        if(fuzz.ratio(fin_textt[i],fin_textt[j])<90):
            continue
        else:
            dup.append((fin_textt[i],fin_textt[j]))#all reviews which are very similar and have a value 90 and above

The time taken to give an output when the above code is run on a 10k dataset is awfully large. Please help me optimize this and boost the performance.

asked Sep 18 '20 at 06:07

moonface16

You can't do much without changing your requirements. Are there duplicates in those lists? – Selcuk Sep 18 '20 at 06:20
try to use multi thread – Hooting Sep 18 '20 at 06:25
Yes, dup contains a list of duplicate reviews. – moonface16 Sep 18 '20 at 06:26
Can you please tell me how to use multi thread in this case? – moonface16 Sep 18 '20 at 06:32
You could try [Python Fuzzy matching strings in list performance](https://stackoverflow.com/questions/56040817/python-fuzzy-matching-strings-in-list-performance) whose answer by Atendra vectorized the computation. There may be an may be CPU memory issue with 10k datasets. – DarrylG Sep 18 '20 at 06:35

Performance optimization when using two for loops for list comparison

0 Answers0