Pythonic way of polling a dictionary - using the key's value once it exists

Question

I have a working solution to this question, it just doesn't feel very pythonic. I am working in Python 2.7 and, thus, cannot use Python 3 solutions.

I have a dictionary that is regularly being updated. Eventually a key, let's call it "foo", with a value will appear in the dictionary. I want to keep polling that object and getting that dictionary until the key "foo" appears at which point I want to get the value associated with that key and use it.

Here is some psuedo code that is functioning right now:

polled_dict = my_object.get_dict()
while('foo' not in polled_dict.keys()):
    polled_dict = my_object.get_dict()
fooValue = polled_dict['foo']

Let me emphasize that what the code is doing right now works. It feels gross but it works. A potential saolution I came up with is:

fooValue = None
While fooValue is None:
    polled_dict = my_object.get_dict()
    fooValue = polled_dict.get('foo')

This also works but it only seems a tiny bit better. Instead of calling polled_dict.get('foo') twice once it shows up in the dict(the key is accessed during the while loop and again on exiting the while loop) we only call it once. But, honestly, it doesn't seem much better and the gains are minimal.

As I look over the other solutions I've implemented I see that they're just different logical permutations of the two above examples (a not in a different place or something) but nothing feels pythonic. I seems like there would be an easy, cleaner way of doing this. Any suggestions? If not, is either of the above better than the other?

EDIT A lot of answers are recommending I override or otherwise change the dictionaries that the code is polling from. I agree that this would normally be a great solution but, to quote from some of my comments below:

"The code in question needs to exist separately from the API that updates the dictionary. This code needs to be generic and access the dictionary of a large number of different types of objects. Adding a trigger would ultimately require completely reworking all of those objects (and would not be nearly as generic as this function needs to be) This is grossly simplified obviously but, ultimately, I need to check values in this dict until it shows up instead of triggering something in the object. I'm unconvinced that making such a wide reaching and potentially damaging change is a pythonic solution(though should the API be rewritten from the ground up this will definitely be the solution and for something that does not need to be separated/can access the API this is definitely the pythonic solution.)"

Membership testing works for `dict` directly, no need to call `.keys()`; `while 'foo' not in polled_dict: polled_dict = my_object.get_dict()`. — Martijn Pieters, Jun 09 '14 at 18:55
I agree with your feeling about something being not quite right. A busy-loop is an expensive way to wait. Better to be notified than to continually poll, but I'm not sure how I would construct that. — Steven Rumbalski, Jun 09 '14 at 18:57
Are you using threading? Why not just trigger something when you add the relevant entry, rather than polling for it until it appears? — user2357112, Jun 09 '14 at 18:57
As you said loop is not CPU friendly and is pointless in this case. It executes extremely fast. It's also not safe either. Normally you would want to have a database, even a lightweight reddis should do. If you want, just do an event like http://stackoverflow.com/questions/5186520/python-property-change-listener-pattern — CppLearner, Jun 09 '14 at 19:00
The code in question needs to exist separately from the API that updates the dictionary. This code needs to be generic and access the dictionary of a large number of different types of objects. Adding a trigger would ultimately require completely reworking all of those objects (and would not be nearly as generic as this function needs to be) This is grossly simplified obviously but, ultimately, I need to check values in this dict until it shows up instead of triggering something in the object. — Nahkki, Jun 09 '14 at 19:01
@Nahkki: Given those constraints it seems your only solution is to continuously poll. — Steven Rumbalski, Jun 09 '14 at 19:02

djhoese · Answer 1 · 2014-06-09T19:42:16.607

1

If your object is modifying the dictionary in place then you should only need to get it once. Then you and your object have a pointer to the same dictionary object. If you need to stick with polling then this is probably the cleanest solution:

polled_dict = my_object.get_dict()
while 'foo' not in polled_dict:
    pass # optionally sleep
fooValue = polled_dict['foo']

The best overall way of doing this would be to push some type of event through a pipe/socket/thread-lock in some way.

edited Jun 09 '14 at 19:42

answered Jun 09 '14 at 19:00

djhoese

3,567
1
27
45

Unfortunately looking at the get_dict() code it looks like it is rebuilding the dictionary every time it is called instead of modifying the dictionary in place. While this is currently out of scope for the changes I am making to the dictionary polling code, I'm going to look into changing that behaviour in the future so that dictionaries are modified in place instead of static/recreated(as this is something that can be changed as opposed to all of the objects triggering a variety of methods.) – Nahkki Jun 09 '14 at 19:13

score 1 · Accepted Answer · answered Jun 09 '14 at 19:31

1

You could always do something like subclass dict.

This is completely untested, but something to the effect of:

class NoisyDict(dict):
    def __init__(self, *args, **kwargs):
        self.handlers = {}
        #Python 3 style
        super().__init__(*args, **kwargs)

    def add_handler(self, key, callback):
        self.handlers[key] = self.handlers.get(key, [])
        self.handlers[key].append(callback)

    def __getitem__(self, key):
        for handler in self.handlers.get(key, []):
            handler('get', key, super().__getitem__(key))
        return super().__getitem__(key)

    def __setitem__(self, key, value):
        for handler in self.handlers.get(key, []):
            handler('set', key, value)
        return super().__setitem(value)

Then you could do

d = NoisyDict()

d.add_handler('spam', print)

d['bar'] = 3
d['spam'] = 'spam spam spam'

answered Jun 09 '14 at 19:31

Wayne Werner

49,299
29
200
290

This is an interesting solution and, certainly, adding a trigger in the dictionary would be a good solution in most circumstances. But such a solution would both require modifying a ton of different classes(and methods) and potentially break all sorts of things. I'm unconvinced that making such a wide reaching and potentially damaging change is a pythonic solution(though should the API be rewritten from the ground up this solution will definitely come up as an option.) – Nahkki Jun 09 '14 at 19:39
I agree about the wide reaching change probably not being so Pythonic. However, I'd also suggest that using the right tool for the job is extremely Pythonic - and the problem that you have doesn't seem to be solved very well by a dictionary. Granted, it seems like you're looking at a fairly large refactor of your code base... but it might be worthwhile. Without seeing your code/architecture the best I can do is take shots in the dark. The solution you're asking for is something that sounds like [select](https://docs.python.org/3/library/select.html) but as it applies to a dict – Wayne Werner Jun 10 '14 at 13:49

score 1 · Answer 3 · answered Jun 09 '14 at 19:33

1

Fun with generators:

from itertools import repeat
gen_dict = (o.get_dict() for o in repeat(my_object))
foo_value = next(d['foo'] for d in gen_dict if 'foo' in d)

answered Jun 09 '14 at 19:33

otus

5,572
1
34
48

Can you explain how this would work a bit more? I don't understand how it repolls the dictionary if foo_value isn't found. It looks like it just runs gen_dict forever which is not what I want because the object exists 'forever'. – Nahkki Jun 09 '14 at 19:45
@Nahkki, each time you iterate the generator `gen_dict`, it polls for a new version of the dict. In the next expression, `next` iterates the dicts returned by the generator until it finds one that has `'foo'` in it, then returns the value for that key in that dict. – otus Jun 10 '14 at 05:32

score 1 · Answer 4 · answered Jun 09 '14 at 19:38

Is it not possible to do something like this? (obviously not thread safe) The only catch is that the method below does not catch dictionary initialization via construction. That is it wouldn't catch keys added when the dictionary is created; eg MyDict(watcher=MyWatcher(), a=1, b=2) - the a and b keys would not be caught as added. I'm not sure how to implement that.

class Watcher(object):
    """Watches entries added to a MyDict (dictionary).  key_found() is called
    when an item is added whose key matches one of elements in keys.
    """
    def __init__(self, *keys):
        self.keys = keys

    def key_found(self, key, value):
        print key, value


class MyDict(dict):

    def __init__(self, *args, **kwargs):
        self.watcher = kwargs.pop('watcher')
        super(MyDict, self).__init__(*args, **kwargs)

    def __setitem__(self, key, value):
        super(MyDict, self).__setitem__(key, value)
        if key in self.watcher.keys:
              self.watcher.key_found(key, value)


watcher = Watcher('k1', 'k2', 'k3')
d = MyDict(watcher=watcher)
d['a'] = 1
d['b'] = 2
d['k1'] = 'k1 value'

You would have to override more methods than `__setitem__` (like `setdefault` and `update`). — Steven Rumbalski, Jun 09 '14 at 19:43
@StevenRumbalski Good point, this question / answer explains how to do that: http://stackoverflow.com/questions/2060972/subclassing-python-dictionary-to-override-setitem — jaime, Jun 09 '14 at 20:30

score 1 · Answer 5 · answered Jun 10 '14 at 07:08

Maybe Try/Except would be considered more 'Pythonic'?

A sleep statement in the while loop will stop it consuming all your resources as well.

polled_dict = my_object.get_dict()
while True:
    time.sleep(0.1)
    try: 
        fooValue = polled_dict['foo']
        return (foovalue) # ...or break
    except KeyError:
        polled_dict = my_object.get_dict()

score 0 · Answer 6 · answered Jun 09 '14 at 19:52

I think a defaultdict is great for this kind of job.

from collections import defaultdict

mydeafultdict = defaultdict(lambda : None)
r = None

while r is None:
    r = mydeafultdict['foo']

a defaultdict works just like a regular dictionary, except when a key doesn't exist, it calls the function supplied to it in the constructor to return a value. In this case, I gave it a lambda that just returns None. With this, you can keep trying to get foo, when there is a value associated with it, it will be returned.

You can implement the same with the built-in `get` method of dictionaries. It will return `None` as a default value if a key doesn't exist. — Burhan Khalid, Jun 09 '14 at 19:55

Pythonic way of polling a dictionary - using the key's value once it exists

6 Answers6