How to remove many elements in a very big dict?

Question

I have a very large dict, and I want to del many elements from it. Perhaps I should do this:

new_dict = { key:big_dict[key] for key in big_dict if check(big_dict[key] }

However, I don't have enough memory to keep both old_dict and new_dict in RAM. Is there any way to deal?

Add:

I can't del elements one by one. I need to do a test for the values to find which elements I want to del.

I also can't del elements in a for loop like:

for key in dic: 
    if test(dic(key)): 
        del dic[key]

It case a error... Can't change len(dic) in the loop...

My God... I even can't make a set to remember keys to del, there are too much keys...

I see, if dict class don't have a function to do this, perhaps the only way to do this is to bug a new computer...

Possible duplicate of [Delete an element from a dictionary](http://stackoverflow.com/questions/5844672/delete-an-element-from-a-dictionary) — l'L'l, May 23 '16 at 06:48

score 2 · Accepted Answer · answered May 23 '16 at 08:10

Here are some options:

Make a new 'dict' on disk, for which pickle and shelve may be helpful.
Iterate through and build up a list of keys until it reaches a certain size, delete those, and then repeat the iteration again, allowing you to make a bigger list each time.
Store the keys to delete in terms of their index in .keys(), which can be more memory efficient. This is OK as long as the dictionary is not modified between calls to .keys(). If about half of the elements are to be deleted, do this with a binary sequeunce (1 = delete, 0 = keep). If a vast majority of elements are to be deleted (or not deleted) store the appropriate keys as integers in a list.

score 1 · Answer 2 · answered May 23 '16 at 06:06

1

You could try iterating through the dictionary and deleting the element that you do not require by

del big_dict[key]

This way you wouldn't be making copies of the dictionary.

answered May 23 '16 at 06:06

Altronicx

I can't del elements one by one. And I also can't del elements in a for loop just like: for key in dic: if test(dic(key)): del key, it case a error... I even can't make a set to remember what to del, there are too mush keys and I have little memory... Perhaps I should get a stronger computer. – Jiahao Xu May 23 '16 at 07:07
What are the specifications of the computer that you are using? – Altronicx May 24 '16 at 20:09

score 0 · Answer 3 · edited May 23 '17 at 11:48

0

You can use

big_dict.pop("key", None)

edited May 23 '17 at 11:48

Community

answered May 23 '16 at 06:06

Rahul

3 Answers3