Incorrect format (No newline chars) when loading YAML with embedded JSON

Question

I am trying to parse a YAML input from a file:

root: {
   children : { key: "test-key", version: "{{ test_version | default( '1.0.0-SNAPSHOT' ) }}"}
}

I am using ruamel.yaml, the section of code that makes the load is configured to preserve quotes and then I am adding manually a new entry:

yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
yaml.width = 4096
yaml.indent(sequence=4, offset=2)

with open(yml_file, 'r') as file:
   print("Modifying file: '%s'..." % str(file))
   data = yaml.load(file)

data['root'][new_project_name.lower()] = {'key': "%s" % new_project_name.lower(),
                                                          'test_version': "{{ %s_version | default(\'1.0.0-SNAPSHOT\') }}"
                                                                     % new_project_name.lower()}

 with open(yml_file, 'w') as file:
       yaml.dump(data, file)

The thing is that when the file gets written with the new entry, I am getting everything in the same line, so it seems not to preserve the new lines (CR LF), (it seems to be loading it without them even) do you know if there is any way to preserve them?.

output is (everything in the same line):

root: {children : { key: "test-key", version: "{{ test_version | default( '1.0.0-SNAPSHOT' ) }}"}}

Check the question https://stackoverflow.com/questions/24418449/pretty-output-with-pyyaml. Answers provide possible solutions with `pyyaml` and `ruamel.yaml` — , Sep 18 '18 at 12:45
Your output is fake, it cannot be generated with any version of `ruamel.yaml` that I am aware of. There is no part of the code that generates a space between the mapping indicator and the key, and certainly no code that sometimes does so (before `key:`) and sometimes doesn't (before `children:`). *Always* provide working code so we can see what you really do (wrong), there are just a few lines missing to be able to cut and paste, why make things incomplete? — Anthon, Sep 18 '18 at 17:28
You are right, I needed to edit the output to show you the concept, so probably I made a mistake, sadly I can`t use the original code. Anyway, the output, is the same without the spaces, as you well said. — galegofer, Sep 19 '18 at 08:12

score 0 · Accepted Answer · answered Sep 18 '18 at 17:25

ruamel.yaml does not preserve comments nor does it preserve spacing within flow style. If you care about layout, so that your YAML is more easy for humans to read, you should be using flow style at the maximum for leaf nodes if at all. That is the default dump style when using YAML(typ='fast').

When you have nested flow-style as with your input, the flow style on these nodes is preserved, and standard formatting is done (except for wrapping if the line becomes to large, everything on one line).

Setting the indent level only affects block style constructs.

You should change the input to only leaf-node flow-style, for better readability:

root: 
   children: {key: "test-key", version: "{{ test_version | default( '1.0.0-SNAPSHOT' ) }}"}

This loads to the same data structure as your input does.

With that you can now do:

import sys
import ruamel.yaml

yml_file = 'input.yaml'
new_project_name = 'NPN'


yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
yaml.width = 4096

with open(yml_file, 'r') as file:
   print("Modifying file: '%s'..." % str(file))
   data = yaml.load(file)

npn_lower = new_project_name.lower()

data['root'][npn_lower] = m = ruamel.yaml.comments.CommentedMap([
    ('key',  "%s" % npn_lower),
    ('test_version', "{{ %s_version | default(\'1.0.0-SNAPSHOT\') }}" % npn_lower)
])
m.fa.set_flow_style()

with open('output.yaml', 'w') as fp:
    yaml.dump(data, fp)

which prints:

Modifying file: '<_io.TextIOWrapper name='input.yaml' mode='r' encoding='UTF-8'>'...

and has as output.yaml:

root:
  children: {key: "test-key", version: "{{ test_version | default( '1.0.0-SNAPSHOT' ) }}"}
  npn: {key: npn, test_version: "{{ npn_version | default('1.0.0-SNAPSHOT') }}"}

Things to note:

add a CommentedMap, as on a normal dict you cannot individually set flow style, and you need to as this is no longer a nested flow-style. The elements are added as a list of tuples, as with older versions of Python the key order is not guaranteed to be the same as in your input. You also create an empty CommentedMap() and add the keys/value pairs one at a time.
during trying things out (and with code presented here) it is always as bad idea to change the input file, as for every test-run your input has to be reverted.

Incorrect format (No newline chars) when loading YAML with embedded JSON

1 Answers1