PyYaml combining two yaml files

Question

I'm trying to set up a system where I have a couple (possibly more) yaml files that will be used for configuration. I want to be able to reference things in one file in another.

I know that YAML doesn't allow this.

My plan, I think, is to combine the two YAML files, and then treat it as a single file. I'm pretty sure that I could either cat the two files together, create a temp file, and read that one as YAML, or read the files as text, concatenate them and THEN parse the string.

However, I feel that there should be a better way to do this. Is there?

Answer 1

The only way of referencing in YAML is to use & (anchors) and * (aliases). For these to work they have to be in the same YAML document. The following will not work (this is based on the merge key feature, but normal object referencing has the same limitation):

import ruamel.yaml

yaml_str = """\
a: &BASE { x: 1, y: 2}
---
b:
  << : *BASE
  z: 3
"""

for data in ruamel.yaml.load_all(yaml_str):
    print(data)

throws a composer error that "BASE" is not found. Remove the --- document separator and everything is fine.

So in principle concatenating two documents could work. Loading the document with the alias separately cannot be done without concatenating it with the one that contains it anchor.

Additionally the caveat is that all documents have to have either a mapping or sequence at the toplevel. If would combine a sequence:

- &BASE a
- b

with a mapping:

c: 1
d: *BASE

the result will not be loadable.

---

As indicated, if the toplevel type is the same for all files, you cannot load the YAML files and combine them in memory. I.e. given the example in the merge key documentation split into 1.yaml:

- &CENTER { x: 1, y: 2 }
- &LEFT { x: 0, y: 2 }
- &BIG { r: 10 }
- &SMALL { r: 1 }

2.yaml:

# Explicit keys
-
  x: 1
  y: 2
  r: 10
  label: center/big

3.yaml:

# Merge one map
-
  << : *CENTER
  r: 10
  label: center/big

4.yaml:

# Merge multiple maps
-
  << : [ *CENTER, *BIG ]
  label: center/big    

5.yaml:

# Override
-
  << : [ *BIG, *LEFT, *SMALL ]
  x: 1
  label: center/big

You cannot use load() on the individual YAML files and combine them:

import ruamel.yaml
import glob

data = []
for file_name in sorted(glob.glob('*.yaml')):
    data.append(ruamel.yaml.load(open(file_name)))
print(ruamel.yaml.dump(data, allow_unicode=True))

(the above which would work if 2.yaml, etc. didn't have the aliases)

If you don't want to concatenate the files outside of your program, you can use this class:

class CombinedOpenForReading(object):
    def __init__(self, file_names):
        self._to_do = file_names[:]
        self._fp = None

    def __enter__(self):
        return self

    def __exit__(self, exception_type, exception_value, exception_traceback):
        if self._fp:
            self._fp.close()

    def read(self, size=None):
        res = ''
        while True:
            if self._fp is None:
                if not self._to_do:
                    return res
                else:
                    self._fp = open(self._to_do.pop(0))
            if size is None:
                data = self._fp.read()
            else:
                data = self._fp.read(size)
            if size is None or not data:
                self._fp.close()
                self._fp = None
            res += data
            if size is None:
                continue
            size -= len(data)
            if size == 0:
                break
        return res

to do:

import ruamel.yaml
import glob

with CombinedOpenForReading(sorted(glob.glob('*.yaml'))) as fp:
    data = ruamel.yaml.round_trip_load(fp)
assert data[6]['r'] == 10
print(ruamel.yaml.dump(data, Dumper=ruamel.yaml.RoundTripDumper))

to get:

- &CENTER {x: 1, y: 2}
- &LEFT {x: 0, y: 2}
- &BIG {r: 10}
- &SMALL {r: 1}
# Explicit keys
- x: 1
  y: 2
  r: 10
  label: center/big
# Merge one map
- <<: *CENTER
  r: 10
  label: center/big
# Merge multiple maps
- <<: [*CENTER, *BIG]
  label: center/big
# Override
- <<: [*BIG, *LEFT, *SMALL]
  x: 1
  label: center/big

(You have to hand in the files in the right order, hence the sort. And make sure that you have newlines at the end of your files, otherwise you might get unexpected errors.)

Answer 2

I think that this is simpler than @Anthon's. It may not be as complete, but I think it's all that I need...

def merge(fList):
    ''' 
    Takes a list of yaml files and loads them as a single yaml document.
    Restrictions:
    1) None of the files may have a yaml document marker (---)
    2) All of the files must have the same top-level type (dictionary or list)
    3) If any pointers cross between files, then the file in which they are defined (&) must be 
    earlier in the list than any uses (*).
    '''

    if not fList:
        #if flist is the empty list, return an empty list. This is arbitrary, if it turns out that
        #an empty dictionary is better, we can do something about that.
        return []

    sList = []
    for f in fList:
        with open(f, 'r') as stream:
            sList.append(stream.read())
    fString = ''
    for s in sList:
        fString = fString + '\n'+ s

    y = yaml.load(fString)

    return y

comments welcome.