Popular data exchange format that preserves input order even to represent unordered/unused elements?

Question

I've recently built a django JSON field, YAML field, Python field to explore ways to store/edit arbitrarily complex hierarchies of data via a CMS.

JSON, YAML, and Python can all be human formatted to be intuitive, but as soon as I coerce the text to python and back, it creates a new object which doesn't have ordering.

Is there a portable data scheme that transparently preserves input order even if the data isn't actually ordered simply to reconstruct the original input however "the humans" decided to build it?

Take for example python/json:

[{ 
   'title': 'First Data Entry!', # intuitive to have certain elements at top
   'is_active': True, 
   'data': 'data here'
   'a_list': [1, 2, 3],
},
{ 
   'title': 'Some Data Entry Here!', 
   'is_active': False, 
   'data': 'data here'
   'a_list': [1, 2, 3],
}] 

Or YAML:

I can set up my data in a way that's instantly recognizable to my eye e.g. titles as the first line by convention.

As soon as I parse this into a python object, modify it, and convert it to a YAML/JSON/Python again, I of course no longer have comments or the ordering.

I'm curious if this is a common issue or if this is a wheel I need to invent. It seems possible with OrderedDict and writing a serializer (which I've never done...).

My current solution is to use YAML as a raw text field which is readonly - it won't try to write python objects to YAML to preserve style.

Answer 1

Normally, dictionaries have no ordering, thats ther reason of change of orders...

One possible way might be using sorting functions on the dictionary (and then transform it to a OrderedDict... Or maybe you do not even need OrderedDict at all). But that orders as ascending/descending, so it might not help...

Final solution might be looing all dictionary and picking the keys one by one as you wish, and put them to your OrderedDict

Here is a good SO question about dictionary ordering...

Answer 2

In python folded style (>) scalars are not preserved, and neither is the ordering.

If you can live with dropping the folded style, you can use the python package¹ ruamel.yaml which is a derivative of PyYAML and supports round trip preservation of comments and roundtrip preservation of ordering of the keys of mappings:

from __future__ import print_function

import ruamel.yaml

inp = """\
features:
  show: true
  items:
    - widget: full_width.html  # full width 1
      title: Some Title
      description: >
          Foobar.
      vimeo_id: 20913
      zoom_image: some_url.png
    - widget: 3x_container.html
      items:
          - widget: 3x.html
            title: Some Widget Title
            image: 'foobar.png'
            description: >
                Some Description.
          - widget: 3x.html
            title: Some new title here
            image: ajax_uploads/png1_2.png
            description: >
                Some Description.
"""

code = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)

res = ruamel.yaml.dump(code, Dumper= ruamel.yaml.RoundTripDumper)
print(res, end='')

with result:

features:
  show: true
  items:
  - widget: full_width.html    # full width 1
    title: Some Title
    description: 'Foobar.

      '
    vimeo_id: 20913
    zoom_image: some_url.png
  - widget: 3x_container.html
    items:
    - widget: 3x.html
      title: Some Widget Title
      image: foobar.png
      description: 'Some Description.

        '
    - widget: 3x.html
      title: Some new title here
      image: ajax_uploads/png1_2.png
      description: 'Some Description.

        '

Which is stable output after the first round-trip.

Instead of normal list and dict objects the code consists of wrapped versions² on which the comments attached.

¹ _{Install with pip install ruamel.yaml. Works on Python 2.6/2.7/3.3+}
² _{ordereddict isused in case of a mapping, to preserve ordering}