Use Python for Creating JSON

Question

I want to use Python for creating JSON.

Since I found no library which can help me, I want to know if it's possible to inspect the order of the classes in a Python file?

Example

# example.py
class Foo:
    pass

class Bar:
    pass

If I import example, I want to know the order of the classes. In this case it is [Foo, Bar] and not [Bar, Foo].

Is this possible? If "yes", how?

Background

I am not happy with yaml/json. I have the vague idea to create config via Python classes (only classes, not instantiation to objects).

Answers which help me to get to my goal (Create JSON with a tool which is easy and fun to use) are welcome.

Answer 1

The inspect module can tell the line numbers of the class declarations:

import inspect

def get_classes(module):
    for name, value in inspect.getmembers(module):
        if inspect.isclass(value):
            _, line = inspect.getsourcelines(value)
            yield line, name

So the following code:

import example

for line, name in sorted(get_classes(example)):
    print line, name

Prints:

2 Foo
5 Bar

Answer 2

First up, as I see it, there are 2 things you can do...

1. Continue pursuing to use Python source files as configuration files. (I won't recommend this. It's analogous to using a bulldozer to strike a nail or converting a shotgun to a wheel)

2. Switch to something like TOML, JSON or YAML for configuration files, which are designed for the job.

   Nothing in JSON or YAML prevents them from holding "ordered" key-value pairs. Python's dict data type is unordered by default (at least till 3.5) and list data type is ordered. These map directly to object and array in JSON respectively, when using the default loaders. Just use something like Python's OrderedDict when deserializing them and voila, you preserve order!

---

With that out of the way, if you really want to use Python source files for the configuration, I suggest trying to process the file using the ast module. Abstract Syntax Trees are a powerful tool for syntax level analysis.

I whipped a quick script for extracting class line numbers and names from a file.

You (or anyone really) can use it or extend it to be more extensive and have more checks if you want for whatever you want.

import sys
import ast
import json


class ClassNodeVisitor(ast.NodeVisitor):

    def __init__(self):
        super(ClassNodeVisitor, self).__init__()
        self.class_defs = []

    def visit(self, node):
        super(ClassNodeVisitor, self).visit(node)
        return self.class_defs

    def visit_ClassDef(self, node):
        self.class_defs.append(node)


def read_file(fpath):
    with open(fpath) as f:
        return f.read()


def get_classes_from_text(text):
    try:
        tree = ast.parse(text)
    except Exception as e:
        raise e

    class_extractor = ClassNodeVisitor()

    li = []
    for definition in class_extractor.visit(tree):
        li.append([definition.lineno, definition.name])

    return li


def main():
    fpath = "/tmp/input_file.py"

    try:
        text = read_file(fpath)
    except Exception as e:
        print("Could not load file due to " + repr(e))
        return 1

    print(json.dumps(get_classes_from_text(text), indent=4))


if __name__ == '__main__':
    sys.exit(main())

---

Here's a sample run on the following file:

input_file.py:

class Foo:
    pass


class Bar:
    pass

Output:

$ py_to_json.py input_file.py
[
    [
        1,
        "Foo"
    ],
    [
        5,
        "Bar"
    ]
]

---

If I import example,

If you're going to import the module, the example module to be on the import path. Importing means executing any Python code in the example module. This is a pretty big security hole - you're loading a user-editable file in the same context as the rest of the application.

Answer 3

(Moving my comments to an answer)

That's a great vague idea. You should give Figura a shot! It does exactly that.

(Full disclosure: I'm the author of Figura.)

I should point out the order of declarations is not preserved in Figura, and also not in json.

I'm not sure about order-preservation in YAML, but I did find this on wikipedia:

... according to the specification, mapping keys do not have an order

It might be the case that specific YAML parsers maintain the order, though they aren't required to.

Answer 4

I'm assuming that since you care about preserving class-definition order, you also care about preserving the order of definitions within each class.

It is worth pointing out that is now the default behavior in python, since python3.6.

Aslo see PEP 520: Preserving Class Attribute Definition Order.

Answer 5

You can use a metaclass to record each class's creation time, and later, sort the classes by it.

This works in python2:

class CreationTimeMetaClass(type): 
    creation_index = 0
    def __new__(cls, clsname, bases, dct):
        dct['__creation_index__'] = cls.creation_index
        cls.creation_index += 1
        return type.__new__(cls, clsname, bases, dct)

__metaclass__ = CreationTimeMetaClass

class Foo: pass
class Bar: pass

classes = [ cls for cls in globals().values() if hasattr(cls, '__creation_index__') ]
print(sorted(classes, key = lambda cls: cls.__creation_index__))

Answer 6

The standard json module is easy to use and works well for reading and writing JSON config files.

Objects are not ordered within JSON structures but lists/arrays are, so put order dependent information into a list.

I have used classes as a configuration tool, the thing I did was to derive them from a base class which was customised by the particular class variables. By using the class like this I did not need a factory class. For example:

from .artifact import Application
class TempLogger(Application): partno='03459'; path='c:/apps/templog.exe'; flag=True
class GUIDisplay(Application): partno='03821'; path='c:/apps/displayer.exe'; flag=False

in the installation script

from .install import Installer
import app_configs

installer = Installer(apps=(TempLogger(), GUIDisplay()))
installer.baseline('1.4.3.3475')
print installer.versions()
print installer.bill_of_materials()

One should use the right tools for the job, so perhaps python classes are not the right tool if you need ordering.

Another python tool I have used to create JSON files is Mako templating system. This is very powerful. We used it to populate variables like IP addresses etc into static JSON files that were then read by C++ programs.

Answer 7

I'm not sure if this is answers your question, but it might be relevant. Take a look at the excellent attrs module. It's great for creating classes to use as data types.

Here's an example from glyph's blog (creator of Twisted Python):

import attr
@attr.s
class Point3D(object):
    x = attr.ib()
    y = attr.ib()
    z = attr.ib()

It saves you writing a lot of boilerplate code - you get things like str representation and comparison for free, and the module has a convenient asdict function which you can pass to the json library:

>>> p = Point3D(1, 2, 3)
>>> str(p)
'Point3D(x=1, y=2, z=3)'
>>> p == Point3D(1, 2, 3)
True
>>> json.dumps(attr.asdict(p))
'{"y": 2, "x": 1, "z": 3}'

The module uses a strange naming convention, but read attr.s as "attrs" and attr.ib as "attrib" and you'll be okay.

Answer 8

Just touching the point about creating JSON from python. there is an excellent library called jsonpickle which lets you dump python objects to json. (and using this alone or with other methods mentioned here you can probably get what you wanted)