0

In a python simulation code, code outputs are located in a nested-dict. I'm interested to log this nested dict to file at the end of each iteration, and having trouble identifying the best practice approach. In a current method, the nest_dict is logged by passing it to a logger like:

self.logger.info(self.input_dict)

But this produces a perhaps wasteful output file, as the dictionary is represented as it would appear with

print(self.input_dict)

So my related questions are:

  1. Are there any best practices around logging nested dictionaries?
  2. Is logging.info the correct method to use or are there dedicated methods in python for logging data, versus logging events?
Paul Fleming
  • 414
  • 2
  • 11

1 Answers1

1

By "wasteful" I assume (perhaps falsely) that would prefer saving it in a binary format. Traditionally, people have used pickle for binary, Python-specific object serialization.

As for how to use it, it's very straightforward (adapted from an official tutorial):

To write:

import pickle

pickle_out = open(r'C:\pathToDict\dictfile', 'wb')
pickle.dump(self.input_dict, pickle_out)
pickle_out.close()

To read:

import pickle

pickle_in = open(r'C:\pathToDict\dictfile', 'rb')
self.input_dict = pickle.load(pickle_in)
pickle_in.close()

To answer your specific questions:

  1. Python does serialization of nested data structures well.
    • pickle has traditionally been more powerful in terms of type coverage, but it is Python-specific and incurs a security risk when one deserializes ("unpickles") data from untrusted sources (this includes the case when someone else can alter your data).
    • JSON (via import json) is a widely known and hugely popular language-interoperable non-binary (text-based) format.
    • Also check out PyYAML. ("YAML is a data serialization format designed for human readability and interaction with scripting languages.")
  2. Usually one logs events and serializes data.
Lover of Structure
  • 1,561
  • 3
  • 11
  • 27
  • Thank you for a detailed response! I'm interested in your point (2). In the solution you provide, wouldn't the result be overwritten each time I hit that block of code? In the solution I have now I can append the whole dictionary to an open file, but was wondering how perhaps to do this in a standard way. But maybe this isn't a standard thing to do. – Paul Fleming Mar 14 '23 at 19:45
  • @PaulFleming Would opening the file with (as applicable) `'ab'` or `'a'` (`a` means "append") work in your case? (I am not sure whether there is a standard way.) – Lover of Structure Mar 15 '23 at 02:28
  • @PaulFleming I think these two questions might address what you are asking about: [How can you Pickle multiple objects in one file?](https://stackoverflow.com/q/42350635/2057969), [Saving and loading multiple objects in pickle file?](https://stackoverflow.com/q/20716812/2057969) – Lover of Structure Mar 15 '23 at 04:18