list comprehension takes too long

Question

I have one of list of dictionaries which I need to compare against another list. I need to only consider items which are present in the 2nd list from list of dictionaries.

In list of dictionaries there can be multiple values against a value in list.

It works fine if both list contains less items. But it takes too long to finish if lists are large. Say my list of dictionaries (myDict) have 500k items, and my 2nd list (filterDict) contains 400k items then process takes too long to complete (it is running after 10 mins) .

myDict = [{'project': "mobileapp1", 'id': "011", 'start_date': '2021-01-01'},
               {'project': "webapp", 'id': "012", 'start_date': '2021-01-01'},
               {'project': "mobileapp2", 'id': "013", 'start_date': '2021-10-01'},
               {'project': "mobileapp2", 'id': "011", 'start_date': '2021-09-01'}]
    
filterDict = ["011",'013']
    
#1 soltion
my_new_dict = [item for item in myDict  if item['id'] in filterDict]
#2 solution
temp = list(filter(lambda d: d['id'] in filterDict, myDict))
print(temp)
print(my_new_dict)

[{'project': 'mobileapp1', 'id': '011', 'start_date': '2021-01-01'}, {'project': 'mobileapp2', 'id': '013', 'start_date': '2021-10-01'}, {'project': 'mobileapp2', 'id': '011', 'start_date': '2021-09-01'}]

Any help would be greatly appreciated.

If you want to test membership, use a `set` rather than a `list`. See for example https://stackoverflow.com/questions/12354515/what-is-the-difference-between-sets-and-lists-in-python — Thierry Lathuille, Nov 21 '21 at 19:01
Concretely, try changing `filterDict = ["011",'013']` to `filterDict = {"011",'013'}`. — Tim Peters, Nov 21 '21 at 19:06
use a tuple or set. it will improve speed. Click [here](https://stackoverflow.com/questions/12354515/what-is-the-difference-between-sets-and-lists-in-python) for examples. — Yash vema, Nov 21 '21 at 19:07
Use a `set` for `filterDict`, so just do `filterDict = set(filterDict)` — juanpa.arrivillaga, Nov 21 '21 at 19:29
Tested with myDict of 400000 and filterDict of 400000 items. It goes down from 845 seconds to 0.08 seconds. — Thomas Weller, Nov 21 '21 at 19:38
Thanks for all the answers. After implementing set (as shown in @Tiim Peter's comment) process takes fraction of second. — user2026504, Nov 22 '21 at 02:22

list comprehension takes too long

0 Answers0