How to use custom dictionary class while loading yaml?

Question

I am currently loading a YAML file like this

 import yaml
 yaml.load('''level0:
                 stuff: string0
                 level1: 
                     stuff: string1
                     level2: ...''')

The code above creates nested dictionaries. Instead of creating nested dictionaries, I want to create nested instances of FancyDict objects.

class FancyDict(collections.MutableMapping):
   def __init__(self, *args, **kwargs):
       for name in kwargs:
          setattr(self, name, kwargs[name])

The section on Constructors, Representer, Resolvers doesn't seem to address this case where I want to globally override the class construction for all dictionaries instead of special tagged ones.

I just need a hook that would be called a object (node?) is created/finalized.
Is there a easy way to do this or should I just traverse the nested dictionaries that a yaml.load returns to me and fix them myself?

Anthon · Accepted Answer · 2016-04-14T18:29:41.383

That hook is not there, the type that is constructed is hard-coded in construct.BaseConstructor.construct_mapping().

The way to solve this is make your own constructor and based on that your own loader, and hand that one in as option for load():

import sys
import collections
import ruamel.yaml as yaml

yaml_str = """\
level0:
  stuff: string0
  level1:
    stuff: string1
    level2: ...
"""

from ruamel.yaml.reader import Reader
from ruamel.yaml.scanner import Scanner
from ruamel.yaml.parser import Parser
from ruamel.yaml.composer import Composer
from ruamel.yaml.constructor import SafeConstructor
from ruamel.yaml.resolver import Resolver
from ruamel.yaml.nodes import MappingNode


class FancyDict(collections.MutableMapping):
    def __init__(self, *args, **kwargs):
        for name in kwargs:
            setattr(self, name, kwargs[name])

    # provide the missing __getitem__, __setitem__, __delitem__, __iter__, and __len__.

class MyConstructor(SafeConstructor):
    def construct_mapping(self, node, deep=False):
        res = SafeConstructor.construct_mapping(self, node, deep)
        assert isinstance(res, dict)
        return FancyDict(**res)


class MyLoader(Reader, Scanner, Parser, Composer, MyConstructor, Resolver):
    def __init__(self, stream, version=None):
        Reader.__init__(self, stream)
        Scanner.__init__(self)
        Parser.__init__(self)
        Composer.__init__(self)
        MyConstructor.__init__(self)
        Resolver.__init__(self)


data = yaml.load(yaml_str, Loader=MyLoader)

When you run this you'll get an error that FancyDict is an abstract class that cannot be instantiated:

TypeError: Can't instantiate abstract class FancyDict with abstract methods __delitem__, __getitem__, __iter__, __len__, __setitem__

I guess your real FancyDict has those implemented.

_{ruamel.yaml is a YAML library that supports YAML 1.2 (I recommend using that, but then I am the author of the package). PyYAML only supports (most of) YAML 1.1. More problematically, it has different constructor.py files for Python2 and Python3, you might not be able to drop in the above code in PyYAML because of that.}

score 0 · Answer 2 · answered Apr 25 '21 at 15:02

I found a solution that actually works on PyYaml.

class Loader(yaml.FullLoader):

    def construct_yaml_map(self, node):
        data = MyDictionaryClass()
        yield data
        value = self.construct_mapping(node)
        data.update(value)

Loader.add_constructor(
    'tag:yaml.org,2002:map',
    Loader.construct_yaml_map
)

The issue with using the solution from this answer is that PyYaml converts the mapping back into a dictionary on the construct_yaml_map function. Just replacing this function in a subclass isn't enough because of the custom add_constructor added for SafeLoader so you can overwrite it to use the new construct_yaml_map for your class.

How to use custom dictionary class while loading yaml?

2 Answers2

Linked