Python throws random KeyError. Could it be lack of computational power?

Question

I keep getting the strangest 'KeyError' when running my script in production. It is strange because in development I am not getting any error.

I created a 'try-except' block to catch the error but everything seems all right with the dictionaries. There are no keys missing. It is strange because sometimes the for loop hangs in the middle of it, or sometimes throws exception (even if it is exactly the same input).

I am using Python 3.4 and the differences between machines are:

Production (AWS' EC2 micro instance) uses only 1 CPU (CPU0) and 1GB RAM, and uses 3 Gunicorn workers.
Development machine has 3 CPUs (CPU0, CPU1, CPU2), 1 GB of RAM and use 3 Guniorn workers (it can be up to 9).

The following is an oversimplification of my code. My real class has 26 attributes and two of them acept dynamically generated HTML SVG code. Also the lenght of a list of dictionaries can have anything from 0 to 100 (in development I use smaller list of dicionaries (i.e. 10)):

class myObject(object):
  self.id = id
  self.alias = alias

def get_me_objects(list_of_dicts):

  try:

    objects_list = []

    print('Entering the faulty for loop...')

    for i in range(len(list_of_dicts)):
        id = list_of_dicts[i]['id']
        alias = list_of_dicts[i]['alias']

        obj = myObject(id, alias)

        print('obj.id:' + str(print(obj.id)))
        print('len(obj.__dict__):' + str(len(obj.__dict__)))

        objects_list.append(obj)

   except:
       print('')
       print('The following exception ocurred:')
       e = sys.exc_info()[0]
       print(e)
       print('obj.id:' + str(obj.id))
       print('len(obj.__dict__):' + str(len(obj.__dict__)))
       print('obj.___dict__:' + str(obj.__dict__))
       
  print('')
  print('Returning objects_list')      
  print(objects_list)
  return objects_list

When I run that with this as input:

get_me_objects([{'id': 1009, 'alias': 'b'}, {'id': 8888, 'alias': 'c'}, {'id': 7778, 'alias': 't'}]

....it tends to give me this output...

Entering the faulty for loop...
obj.id: 1009
len(obj.__dict__): 2
obj.id: 8888
len(obj.__dict__): 2

The following exception ocurred:
<class 'KeyError'>
obj.id: 8888
len(obj.__dict__): 2
obj.___dict__: {'alias': 'c', 'id': 8888}       

Returning objects_list
[<app.views.my_scripts.MyObject object at 0x7f9c4c464198>, <<app.views.my_scripts.MyObject object at 0x7f9c4c46db38>]

...If I retry again it can sometimes give me this output (even with the same list of dictionaries!)...

Entering the faulty for loop...

...Strangely enough, if I retry again sometimes it can give me this output (even with the exactly the same list of dictionaries!):

Entering the faulty for loop...
obj.id: 1009
len(obj.__dict__): 2

The following exception ocurred:
<class 'KeyError'>
obj.id: 1009
len(obj.__dict__): 2
obj.___dict__: {'alias': 'a', 'id': 1009}       

Returning objects_list
[<app.views.my_scripts.MyObject object at 0x7f9c4c464198>]

So I suspect the problem has to do with CPU computer power. Or maybe there's a problem lying somewhere which for the life of me I can't see it. What else can I do to troubleshoot this problem?

UPDATE 1.0

Rob made an important point asking me to remove the try-except block to see the full traceback. The reason I added the try-except block was because Gunicorn was not showing me anything. I'll update my post as soon as I find a way to get the traceback.

UPDATE 2.0

Following Rob advice I check the traceback (obviously!) and found there is a valid KeyError coming from another function being called in from within the for loop. It went unnoticed because first, the name of the function kind of camouflaged with similar variables names, and second, I got seriously distracted because of the 'seemingly random' behaviour I was keep getting from the dictionary:

'In what order does python display dictionary keys?'):

The order has to do with how they work internally and what order they end up in in the hashtable. That in turn depends on the keys hash-value, the order they were inserted, and which Python implementation you are using.

The order is arbitrary (but not random) and it will never be useful to know which order it will be.

To get a sorted list of keys, just use sorted(D), which in your case will return ['a', 'b', 'c'].

My bad. Anyway, thanks everybody for your help!

Please do not catch the exception and show the whole traceback instead. Your exception messages are unhelpful, because it shows `obj` from the previous iteration. — Rob, Mar 25 '16 at 15:07
Fair enough, Rob. I did it that way because Gunicorn was not showing me anything without the except. Let me see what can I do and I'll update my post. — BringBackCommodore64, Mar 25 '16 at 15:10
Can you print the current input at position 'i' before you access the 'id' and 'alias' fields? — Schore, Mar 25 '16 at 15:21
Can you use **logger.exception**? It does tracebacks automatically. — JL Peyret, Mar 25 '16 at 15:33
Your keyerror is on the dict. Use **mydict = list_of_dict[i]** then do the id alias lookup on mydict. When that errors, you'll have guilty dict available to look at in the except clause. — JL Peyret, Mar 25 '16 at 15:45
An unlikely, but no more weird cause than 'CPU power', could be that 'id' is a reserved python keyword. Verrry unlikely, but who knows. — JL Peyret, Mar 25 '16 at 15:51
Thanks, JP Peyret for your help. I also thought about the 'id' reserved keyword. But in my script I don't use 'id'. I should have mentioned that for the sake of clarity. — BringBackCommodore64, Mar 25 '16 at 19:19

score 1 · Answer 1 · edited Jun 20 '20 at 09:12

Following Rob advice I checked the traceback (duh!) and found there is a valid KeyError coming from another function being called in from within the for loop.

The mistake went unnoticed (for me anyway) because first, the name of the function kind of camouflaged with similar variables names, second, I was expecting to see the traceback in the Gunicorn output (but it wasn't happening), and third, I got seriously distracted because of the 'seemingly random' behaviour I was keep getting from the dictionary. A reminder:

'In what order does python display dictionary keys?'

The order has to do with how they work internally and what order they end up in in the hashtable. That in turn depends on the keys hash-value, the order they were inserted, and which Python implementation you are using.

The order is arbitrary (but not random) and it will never be useful to know which order it will be.

To get a sorted list of keys, just use sorted(D), which in your case will return ['a', 'b', 'c'].

Python throws random KeyError. Could it be lack of computational power?

1 Answers1