load variables from in memory setting file

Question

i have settings.py file that looks like:

DEBUG = True
CACHE_IP = '0.0.0.0'
DB_SETTINGS = {
    'SERVERS': ['127.0.0.1:80'],
    'MAX_IDLE_TIME' : 180,
}

this file is stored at external cloud storage.

I have a process the get the url to this file, and load it into memory (not to disk). when loading it into memory it will be string type and looks like:

"DEBUG = True\n CACHE_IP = '0.0.0.0'\n DB_SETTINGS = {\n    'SERVERS': ['127.0.0.1:80'],\n    'MAX_IDLE_TIME' : 180,\n}\n"

How for example i can get the values from this string? i mean, how can i get a dictionary where key=verb name from file, and value = value from file? is there any pythonic way for this?

so my dictionary will look like:

{'DEBUG': True, 'CACHE_IP' = '0.0.0.0', 'DB_SETTINGS': {...}}

I would suggest storing your settings on the cloud as `json` data so you can use python's json parsers to get the data in a dict format. — MCBama, Dec 06 '17 at 18:13
i cant. this file is stored as settings.py. this is settings file for another python app — Eyal Ch, Dec 06 '17 at 18:16
I'll second martineau. The string you provide is rather difficult to take apart because of there is no common format you can use to pick everything apart. If you're able to do so you might also consider changing how the source file is formatted. (e.g. with easy-to-parse `key = val [/n]`) — ikom, Dec 06 '17 at 18:57
If you can't convert it to `json` then you should be able to load the entire file into memory as a file object instead of a string and then iterate over the rows to build a dictionary using `urllib` like in [this](https://stackoverflow.com/questions/32971752/python-read-file-from-a-web-url) question. the `requests` library should be able to do it too. — MCBama, Dec 06 '17 at 19:32

Amit Tripathi · Accepted Answer · 2017-12-07T16:54:38.813

Your requirement is very specific but this kind of requirements do arise. So, I tried to solve this using method below:

In [102]: import ast # Do not use eval(its unsafe)

In [103]: string = "DEBUG = True\n CACHE_IP = '0.0.0.0'\n DB_SETTINGS = {\n    '
     ...: SERVERS': ['127.0.0.1:80'],\n    'MAX_IDLE_TIME' : 180,\n}\n"
     ...:

In [104]: def parse_key_val(string):
     ...:     tmp_dict = {}
     ...:     for r in re.split('(?<![{},])\n (?!{)', string):
     ...:         key, val = r.split(" = ")
     ...:         if '{' in val:
     ...:             val = ast.literal_eval(val)
     ...:         tmp_dict[key] = val
     ...:     return tmp_dict
     ...:

In [105]: parse_key_val(string)
Out[105]:
{'CACHE_IP': "'0.0.0.0'",
 'DB_SETTINGS': {'MAX_IDLE_TIME': 180, 'SERVERS': ['127.0.0.1:80']},
 'DEBUG': 'True'}

This solution works great for the string you provided but it might(or not) fail in some complex cases. But this solution gives a good starting point of how to start. You might need to make some modifications to make it work for complex cases.

Now coming to the program, the important parts here to understand are:

re.split('(?<![{},])\n (?!{)', string)

and

val = ast.literal_eval(val)

The pattern (?<![{},])\n (?!{) in re.split means, split the string on character \n but only if it is not after { or } or , and not before {. This regex is created to not split \n line character in dict strings.

Not the second part val = ast.literal_eval(val) builds on top of the above regex step as above step doesn't break the dict string. The resulting dict part of string after split is:

"DB_SETTINGS = {\n    'SERVERS': ['127.0.0.1:80'],\n    'MAX_IDLE_TIME' : 180,\n}\n"

With ast.literal_eval the task of parsing dict, nested dict and list inside it very easy. That it!

One more thing, do not use eval as one of the solution suggests here, it's a security risk.

You can read more about ast here

score 1 · Answer 2 · answered Dec 06 '17 at 18:21

1

One way is to use the eval function to execute the text.

eval("DEBUG = True\n CACHE_IP = '0.0.0.0'\n DB_SETTINGS = {\n    'SERVERS':       ['127.0.0.1:80'],\n    'MAX_IDLE_TIME' : 180,\n}\n")
settings_dict = {}
settings_dict['DEBUG'] = DEBUG
settings_dict['CACHE_IP'] = DEBUG
settings_dict['DB_SETTINGS'] = DB_SETTINGS

I don't know of a way to 'automate' the dictionary creation.

answered Dec 06 '17 at 18:21

Ram

51
3

1

Note that `eval` just executes code, so if someone puts `import shutil; shutil.rmtree('/home/username')` in there, you've just wiped your home directory! `eval` is a security risk- please be very careful when using it! – Ben Dec 06 '17 at 18:59
eval won't work with the above string. You will get `DEBUG = True SyntaxError: invalid syntax` – Amit Tripathi Dec 06 '17 at 18:59
its not safe to use eval – Eyal Ch Dec 06 '17 at 19:10
oops, eval only evaluates expressions like the dictionary string. As such import statements don't work either. There could still be some creative security compromise. The accepted answer is a better solution. – Ram Dec 06 '17 at 20:47

load variables from in memory setting file

2 Answers2