How to automate saving constants?

Question

I am in the process of running experiments, and ideally, once all of my code is working, the only parameters that will need to be changed will all be present in one file. My initial idea was to store these parameters in a JSON file:

{
    "param1": 1,
    "param2": "string parameter"
}

Where, obviously, I have many more than 2 parameters. This turns out to be a nightmare, as my IDE will not guess any of the parameters, which massively slows down my programming as I generally feel obligated to create local variables for every constant that I need in the current function that I'm working in. This results in a lot of unnecessary code (but local to that function, it is significantly more convenient than trying to index the JSON object).

My next thought was then: store the constants in a file like:

PARAM1 = 1
PARAM2 = 'string parameter'

The problem with this is that I'd like to store the parameters with experimental results so that I can look back to see which parameters were specified to produce those results.

Beyond this, my thought is to use a dataclass (probably one with frozen=True), as those can be converted to a dictionary. However, I do not need access to an instance of the class, just the constants within it.

Another thought is to use a class with static variables:

class ExperimentalMetaData:
    param1 = 1
    param2 = "string parameter"

Which can be converted to a dict with vars(ExperimentalMetaData), except this will contain additional keys that should be popped off before I go about storing the data.

My question is: what is the best way to store constants in python such that they can be saved to a JSON file easily, and also be easily accessed within my code?

What is wrong with having one single dict with all the parameters as key/value elements? — ibarrond, Jan 18 '22 at 16:05
Do you actually need to store them to a *JSON* file, or just any kind of file? Various parts of the question imply different requirements. If JSON is not a requirement, does the file have to be "human readable"? — MisterMiyagi, Jan 18 '22 at 16:07
You can use the .ini format and read/write it using [`configparser`](https://docs.python.org/3/library/configparser.html) (I personally use it in a big project and find it very convenient to run the same code under varying configurations) — Tomerikoo, Jan 18 '22 at 16:08
I had a solution to post with standard dicts, but sadly the question is now closed — ibarrond, Jan 18 '22 at 16:19
@ibarrond I commented this on one of the answers but, using a dictionary requires that I remember the key names and type them out in full which is quite inconvenient and is the point of the question: I'd like my IDE to help me out when writing code, but I'd like to also be able to go look at the constants. My initial hope when posting this question was that I was missing something obvious (and that there was clearly a best solution) but based on the answers I can see this isn't the case, and by StackOverflows policy I suppose it is correct that this was closed. — Kraigolas, Jan 18 '22 at 16:22
But it doesn't! you just need to search on ways to turn a dictionary key/value attributes into something that can be accessed more easily. I'd go with [`namespace`](https://stackoverflow.com/a/50118724/9670056). The advantage? You now have a single object with attributes accessible via `params.param1` syntax, and any IDE will detect the attributes and deal with them (unlike keys in a dict). — ibarrond, Jan 18 '22 at 16:41
@Kraigolas for type hinting support, have you looked into [`TypedDict`](https://www.python.org/dev/peps/pep-0589/)? — rv.kvetch, Jan 19 '22 at 05:45

2e0byo · Answer 1 · 2022-01-18T16:13:42.137

2

Split up your problems.

Storing the data

Serialise it to JSON or YAML (or even csv).

Getting the data

Have a module which reads your json and then sets the right values. Something like:

# constants.py
from json import load

data = load("dump.json")
const1: str = data["const1"]
const2: str = data["const2"]
const3: int = data["const3"]

# some_other_module.py
from constants import const1, const2 # IDE knows what they are

I'd only do this manually with vars in a module for a small (<20) number of vars I needed a lot and didn't want to wrap in some dictionary or the like. Otherwise I'd just use a dict in the module. Pre-populating the dict with keys and None and typehinting it will do the same job of getting autocomplete working.

edited Jan 18 '22 at 16:13

answered Jan 18 '22 at 16:09

2e0byo

5,305
1
6
26

3

There's no use in doing ``const1: str = None``, and it is in fact wrongly typed. Either declare ``const1: str`` ahead, or just use ``const1: str = data["const1"]``. – MisterMiyagi Jan 18 '22 at 16:11
yeah, true: I was trying to make it more explicit, but I'll remove that. `=None` is implicit `Optional[str]` which may or may not be useful. Anyhow I'll update. – 2e0byo Jan 18 '22 at 16:13
This answers my question but it's actually exactly what I was trying to avoid doing as it requires a lot of boilerplate code that changes if the constants file changes. My initial hope when posting this question was that someone would have a bit of a better way to solve the problem, although this is probably the solution I'll use if there isn't. – Kraigolas Jan 18 '22 at 16:15
You can avoid boilerplate in a few ways (e.g. if you need to typehint lots of instances of a few type of vars, wrap them in different objects and hint the object) but at the end of the day if you want to declare things like type about your vars you have to declare them somewhere. If you just want to shut your IDE up without getting hints, dropping everything into a `data` dict or object and accessing its members ought to do the trick. – 2e0byo Jan 18 '22 at 16:18
@2e0byo The problem isn't shutting up my IDE, it's getting it to talk. For the sake of constants type hinting isn't all that important to me: each variable can only ever have one type, and it's obvious from their names. But using a dictionary requires that I remember lengthy variable names and type them out in full to access variables. This is a nightmare when you have 10 variables let alone 30 or 50. – Kraigolas Jan 18 '22 at 16:19

score 2 · Accepted Answer · answered Jan 18 '22 at 16:16

2

If you want to be able to recall different versions of inputs, give them a version

This allows you to create JSON-like input files and keep a collection of parsers which can parse them if you make a breaking change

Here's a very simple example which is more sustainable

class Parser_v1_2(): pass

class Parser_v3_2(): pass

VERSION_PARSER_MAPPING = {
   "1.2": Parser_v1_2,
   "3.2": Parser_v3_2,
}

def parser_map(input_file):
    with open(input_file) as fh:
        input_json = json.load(fh)
    # get version or optionally provide a default
    version = input_json.get("version", "1.0")
    # dynamically select parser
    return VERSION_PARSER_MAPPING[version](input_json)

answered Jan 18 '22 at 16:16

ti7

16,375
6
40
68

I guess it should be `input_json.get("version", "1.2")` – Olvin Roght Jan 18 '22 at 16:18
@OlvinRoght certainly, but _it depends_ .. I can't create a full example, so that is broken in a way that forces the user to create their own change - I actually would recommend _never_ creating a default value like that and instead having a dedicated parser to detect the version of files before a versioning scheme (1.0 just represents their first or current working version here, which won't have a version yet, and so is safe to fall back on .. that behavior, however, is outright dangerous, because new files which forget will be treated as old ones) – ti7 Jan 18 '22 at 16:20

How to automate saving constants?

2 Answers2

Storing the data

Getting the data