0

In my program, I have certain settings that can be modified by the user, saved on the disk, and then loaded when application is restarted. Some these settings are stored as dictionaries. While trying to implement this, I noticed that after a dictionary is restored, it's values cannot be used to access values of another dictionary, because it throws a KeyError: 1 exception.

This is a minimal code example that ilustrates the issue:

import json

motorRemap = {
    1: 3,
    2: 1,
    3: 6,
    4: 4,
    5: 5,
    6: 2,
}

motorPins = {
    1: 6,
    2: 9,
    3: 10,
    4: 11,
    5: 13,
    6: 22
}

print(motorPins[motorRemap[1]]); #works correctly

with open('motorRemap.json', 'w') as fp:
    json.dump(motorRemap, fp)

with open('motorRemap.json', 'r') as fp:
    motorRemap = json.load(fp)

print(motorPins[motorRemap[1]]); #throws KeyError: 1

You can run this code as it is. First print statement works fine, but after the first dictionary is saved and restored, it doesn't work anymore. Apparently, saving/restoring somehow breaks that dictionary.

I have tried saving and restoring with json and pickle libraries, and both produce in the same error. I tried printing values of the first dictionary after it is restored directly ( print(motorRemap[1]), and it prints out correct values without any added spaces or anything. KeyError usually means that the specified key doesn't exist in the dictionary, but in this instance print statement shows that it does exist - unless some underlying data types have changed or something. So I am really puzzled as to why this is happening.

Can anyone help me understand what is causing this issue, and how to solve it?

Justinas Rubinovas
  • 197
  • 14
  • 1
    The keys are restored as strings, not integers. Looks like b/c json requires strings as keys I guess. – topsail Jul 20 '22 at 18:43
  • Why not print `motorRemap`? – Peter Wood Jul 20 '22 at 18:45
  • 1
    See here: [read-a-json-and-convert-the-keys-to-int](https://stackoverflow.com/questions/53050408/read-a-json-and-convert-the-keys-to-int) and maybe here: [pythons-json-module-converts-int-dictionary-keys-to-strings](https://stackoverflow.com/questions/1450957/pythons-json-module-converts-int-dictionary-keys-to-strings) – topsail Jul 20 '22 at 18:47
  • 3
    Use pickle instead of JSON if you want to save Python dictionary format. – Barmar Jul 20 '22 at 18:56
  • @topsail - is there any way I can force json to save and restore dictionary with the original data format? – Justinas Rubinovas Jul 20 '22 at 19:48
  • @Barmar - I have tried that, same issue happens with pickle too. Do I need to pass some special arguments for it to work? – Justinas Rubinovas Jul 20 '22 at 19:48
  • *is there any way I can force json to save and restore dictionary with the original data format* -- see the first link in my first comment - this looks like the most common solution to this problem. Of course you could also re-write your un-serialized dictionary with integer keys again (using a dictionary comprehension). So, just this will do in a pinch: `motorRemap = {int(key):value for (key, value) in motorRemap.items()}` – topsail Jul 20 '22 at 20:01
  • @JustinasRubinovas It works for me when I use pickle. – Barmar Jul 20 '22 at 20:06

2 Answers2

3

What happens becomes clear when you look at what json.dump wrote into motorRemap.json:

{"1": 3, "2": 1, "3": 6, "4": 4, "5": 5, "6": 2}

Unlike Python, json can only use strings as keys. Python, on the other hand, allows many different types for dictionary keys, including booleans, floats and even tuples:

my_dict = {False: 1,
           3.14: 2,
           (1, 2): 3}

print(my_dict[False], my_dict[3.14], my_dict[(1, 2)])
# Outputs '1 2 3'

The json.dump function automatically converts some of these types to string when you try to save the dictionary to a json file. False becomes "false", 3.14 becomes "3.14" and, in your example, 1 becomes "1". (This doesn't work for the more complex types such as a tuple. You will get a TypeError if you try to json.dump the above dictionary where one of the keys is (1, 2).)

Note how the keys change when you dump and load a dictionary with some of the Python-specific keys:

import json

my_dict = {False: 1,
           3.14: 2}

print(my_dict[False], my_dict[3.14])

with open('my_dict.json', 'w') as fp:
    json.dump(my_dict, fp)
    # Writes {"false": 1, "3.14": 2} into the json file

with open('my_dict.json', 'r') as fp:
    my_dict = json.load(fp)
    
print(my_dict["false"], my_dict["3.14"])
# And not my_dict[False] or my_dict[3.14] which raise a KeyError

Thus, the solution to your issue is to access the values using strings rather than integers after you load the dictionary from the json file. print(motorPins[motorRemap["1"]]) instead of your last line will fix your code.

From a more general perspective, it might be worth considering keeping the keys as strings from the beginning if you know you will be saving the dictionary into a json file. You could also convert the values back to integers after loading as discussed here; however, that can lead to bugs if not all the keys are integers and is not a very good idea in bigger scale.

Checkout pickle if you want to save the dictionary keeping the Python format. It is, however, not human-readable unlike json and it's also Python-specific so it cannot be used to transfer data to other languages, missing virtually all the main benefits of json.

If you want to save and load the dictionary using pickle, this is how you would do it:

# import pickle
...
with open('motorRemap.b', 'wb') as fp:
    pickle.dump(motorRemap, fp)

with open('motorRemap.b', 'rb') as fp:
    motorRemap = pickle.load(fp)
...
Filip Müller
  • 1,096
  • 5
  • 18
  • Thank you for the detailed answer, it is clear what the issue is now. However, I did try pickle, and received the same error, meaning that by default, it converted keys to strings too. Do I need to use some special arguments when saving/restoring that dictionary with pickle to make it save the dictionary with the original data format? – Justinas Rubinovas Jul 20 '22 at 19:58
  • Can you provide the code where you tried using pickle? – Filip Müller Jul 20 '22 at 20:07
  • I didn't save that code, but I believe I used this example: https://stackoverflow.com/a/11218504/10243896. I tried it now, and I am still getting KeyError. – Justinas Rubinovas Jul 20 '22 at 20:34
  • I have added an example of how to use `pickle` in this scenario to the answer. I cannot help you more if I don't see the code doesn't work for you. – Filip Müller Jul 20 '22 at 20:51
0

since the keys (integers) from a dict will be written to the json file as strings, we can modify the reading of the json file. using a dict comprehension restores the original dict values:

...
with open('motorRemap.json', 'r') as fp:
    motorRemap = {int(item[0]):item[1] for item in json.load(fp).items()}
...
lroth
  • 367
  • 1
  • 2
  • 4