Extract a subset of key-value pairs from dictionary?

Question

I have a big dictionary object that has several key value pairs (about 16), but I am only interested in 3 of them. What is the best way (shortest/efficient/most elegant) to subset such dictionary?

The best I know is:

bigdict = {'a':1,'b':2,....,'z':26} 
subdict = {'l':bigdict['l'], 'm':bigdict['m'], 'n':bigdict['n']}

I am sure there is a more elegant way than this.

Answer 1

You could try:

dict((k, bigdict[k]) for k in ('l', 'm', 'n'))

... or in Python 3 Python versions 2.7 or later (thanks to Fábio Diniz for pointing that out that it works in 2.7 too):

{k: bigdict[k] for k in ('l', 'm', 'n')}

Update: As Håvard S points out, I'm assuming that you know the keys are going to be in the dictionary - see his answer if you aren't able to make that assumption. Alternatively, as timbo points out in the comments, if you want a key that's missing in bigdict to map to None, you can do:

{k: bigdict.get(k, None) for k in ('l', 'm', 'n')}

If you're using Python 3, and you only want keys in the new dict that actually exist in the original one, you can use the fact to view objects implement some set operations:

{k: bigdict[k] for k in bigdict.keys() & {'l', 'm', 'n'}}

Answer 2

A bit shorter, at least:

wanted_keys = ['l', 'm', 'n'] # The keys you want
dict((k, bigdict[k]) for k in wanted_keys if k in bigdict)

Answer 3

A bit of speed comparison for all mentioned methods:

UPDATED on 2020.07.13 (thx to @user3780389): ONLY for keys from bigdict.

 IPython 5.5.0 -- An enhanced Interactive Python.
Python 2.7.18 (default, Aug  8 2019, 00:00:00) 
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
import numpy.random as nprnd
  ...: keys = nprnd.randint(100000, size=10000)
  ...: bigdict = dict([(_, nprnd.rand()) for _ in range(100000)])
  ...: 
  ...: %timeit {key:bigdict[key] for key in keys}
  ...: %timeit dict((key, bigdict[key]) for key in keys)
  ...: %timeit dict(map(lambda k: (k, bigdict[k]), keys))
  ...: %timeit {key:bigdict[key] for key in set(keys) & set(bigdict.keys())}
  ...: %timeit dict(filter(lambda i:i[0] in keys, bigdict.items()))
  ...: %timeit {key:value for key, value in bigdict.items() if key in keys}
100 loops, best of 3: 2.36 ms per loop
100 loops, best of 3: 2.87 ms per loop
100 loops, best of 3: 3.65 ms per loop
100 loops, best of 3: 7.14 ms per loop
1 loop, best of 3: 577 ms per loop
1 loop, best of 3: 563 ms per loop

As it was expected: dictionary comprehensions are the best option.

Answer 4

interesting_keys = ('l', 'm', 'n')
subdict = {x: bigdict[x] for x in interesting_keys if x in bigdict}

Answer 5

This answer uses a dictionary comprehension similar to the selected answer, but will not except on a missing item.

python 2 version:

{k:v for k, v in bigDict.iteritems() if k in ('l', 'm', 'n')}

python 3 version:

{k:v for k, v in bigDict.items() if k in ('l', 'm', 'n')}

Answer 6

An alternative approach for if you want to retain the majority of the keys while removing a few:

{k: bigdict[k] for k in bigdict.keys() if k not in ['l', 'm', 'n']}

Answer 7

Maybe:

subdict=dict([(x,bigdict[x]) for x in ['l', 'm', 'n']])

Python 3 even supports the following:

subdict={a:bigdict[a] for a in ['l','m','n']}

Note that you can check for existence in dictionary as follows:

subdict=dict([(x,bigdict[x]) for x in ['l', 'm', 'n'] if x in bigdict])

resp. for python 3

subdict={a:bigdict[a] for a in ['l','m','n'] if a in bigdict}

Answer 8

You can also use map (which is a very useful function to get to know anyway):

sd = dict(map(lambda k: (k, l.get(k, None)), l))

Example:

large_dictionary = {'a1':123, 'a2':45, 'a3':344}
list_of_keys = ['a1', 'a3']
small_dictionary = dict(map(lambda key: (key, large_dictionary.get(key, None)), list_of_keys))

PS: I borrowed the .get(key, None) from a previous answer :)

Answer 9

Okay, this is something that has bothered me a few times, so thank you Jayesh for asking it.

The answers above seem like as good a solution as any, but if you are using this all over your code, it makes sense to wrap the functionality IMHO. Also, there are two possible use cases here: one where you care about whether all keywords are in the original dictionary. and one where you don't. It would be nice to treat both equally.

So, for my two-penneth worth, I suggest writing a sub-class of dictionary, e.g.

class my_dict(dict):
    def subdict(self, keywords, fragile=False):
        d = {}
        for k in keywords:
            try:
                d[k] = self[k]
            except KeyError:
                if fragile:
                    raise
        return d

Now you can pull out a sub-dictionary with

orig_dict.subdict(keywords)

Usage examples:

#
## our keywords are letters of the alphabet
keywords = 'abcdefghijklmnopqrstuvwxyz'
#
## our dictionary maps letters to their index
d = my_dict([(k,i) for i,k in enumerate(keywords)])
print('Original dictionary:\n%r\n\n' % (d,))
#
## constructing a sub-dictionary with good keywords
oddkeywords = keywords[::2]
subd = d.subdict(oddkeywords)
print('Dictionary from odd numbered keys:\n%r\n\n' % (subd,))
#
## constructing a sub-dictionary with mixture of good and bad keywords
somebadkeywords = keywords[1::2] + 'A'
try:
    subd2 = d.subdict(somebadkeywords)
    print("We shouldn't see this message")
except KeyError:
    print("subd2 construction fails:")
    print("\toriginal dictionary doesn't contain some keys\n\n")
#
## Trying again with fragile set to false
try:
    subd3 = d.subdict(somebadkeywords, fragile=False)
    print('Dictionary constructed using some bad keys:\n%r\n\n' % (subd3,))
except KeyError:
    print("We shouldn't see this message")

If you run all the above code, you should see (something like) the following output (sorry for the formatting):

Original dictionary:
{'a': 0, 'c': 2, 'b': 1, 'e': 4, 'd': 3, 'g': 6, 'f': 5, 'i': 8, 'h': 7, 'k': 10, 'j': 9, 'm': 12, 'l': 11, 'o': 14, 'n': 13, 'q': 16, 'p': 15, 's': 18, 'r': 17, 'u': 20, 't': 19, 'w': 22, 'v': 21, 'y': 24, 'x': 23, 'z': 25}

Dictionary from odd numbered keys:
{'a': 0, 'c': 2, 'e': 4, 'g': 6, 'i': 8, 'k': 10, 'm': 12, 'o': 14, 'q': 16, 's': 18, 'u': 20, 'w': 22, 'y': 24}

subd2 construction fails:
original dictionary doesn't contain some keys

Dictionary constructed using some bad keys:
{'b': 1, 'd': 3, 'f': 5, 'h': 7, 'j': 9, 'l': 11, 'n': 13, 'p': 15, 'r': 17, 't': 19, 'v': 21, 'x': 23, 'z': 25}

Answer 10

Yet another one (I prefer Mark Longair's answer)

di = {'a':1,'b':2,'c':3}
req = ['a','c','w']
dict([i for i in di.iteritems() if i[0] in di and i[0] in req])

Answer 11

solution

from operator import itemgetter
from typing import List, Dict, Union


def subdict(d: Union[Dict, List], columns: List[str]) -> Union[Dict, List[Dict]]:
    """Return a dict or list of dicts with subset of 
    columns from the d argument.
    """
    getter = itemgetter(*columns)

    if isinstance(d, list):
        result = []
        for subset in map(getter, d):
            record = dict(zip(columns, subset))
            result.append(record)
        return result
    elif isinstance(d, dict):
        return dict(zip(columns, getter(d)))

    raise ValueError('Unsupported type for `d`')

examples of use

# pure dict

d = dict(a=1, b=2, c=3)
print(subdict(d, ['a', 'c']))

>>> In [5]: {'a': 1, 'c': 3}

# list of dicts

d = [
    dict(a=1, b=2, c=3),
    dict(a=2, b=4, c=6),
    dict(a=4, b=8, c=12),
]

print(subdict(d, ['a', 'c']))

>>> In [5]: [{'a': 1, 'c': 3}, {'a': 2, 'c': 6}, {'a': 4, 'c': 12}]

Answer 12

Another way in py3.8+ that avoids None values for keys missing from big_dict uses the walrus:

small_dict = {key: val for key in ('l', 'm', 'n') if (val := big_dict.get(key))}

Answer 13

Using map (halfdanrump's answer) is best for me, though haven't timed it...

But if you go for a dictionary, and if you have a big_dict:

1. Make absolutely certain you loop through the the req. This is crucial, and affects the running time of the algorithm (big O, theta, you name it)
2. Write it generic enough to avoid errors if keys are not there.

so e.g.:

big_dict = {'a':1,'b':2,'c':3,................................................}
req = ['a','c','w']

{k:big_dict.get(k,None) for k in req )
# or 
{k:big_dict[k] for k in req if k in big_dict)

Note that in the converse case, that the req is big, but my_dict is small, you should loop through my_dict instead.

In general, we are doing an intersection and the complexity of the problem is O(min(len(dict)),min(len(req))). Python's own implementation of intersection considers the size of the two sets, so it seems optimal. Also, being in c and part of the core library, is probably faster than most not optimized python statements. Therefore, a solution that I would consider is:

dict = {'a':1,'b':2,'c':3,................................................}
req = ['a','c','w',...................]

{k:dic[k] for k in set(req).intersection(dict.keys())}

It moves the critical operation inside python's c code and will work for all cases.

Answer 14

In case someone wants first few items n of the dictionary without knowing the keys:

n = 5 # First Five Items
ks = [*dikt.keys()][:n]
less_dikt = {i: dikt[i] for i in ks}