Edit parameters in a python file from another python file

Question

I've got a configuration file for an application in the form of a python file containing a dict and I'm running an AWS lambda function to get this conf file from S3, change three variables in it and then push a copy back to S3. Is there any easy way to do this? A coworker said to try Jinja templating, but that seems limited to just HTML files?

Thanks

An example of the python config file is below. I need to change the "alpha" and "cycles" variables

import zutil

alpha = 2.13
cycles = 100


def my_transform(x, y, z):
    v = [x, y, z]
    v = zutil.rotate_vector(v, alpha, 0.0)
    return {'v1': v[0], 'v2': v[1], 'v3': v[2]}

parameters = {
    # units for dimensional quantities
    'units': 'SI',
    # reference state
    'reference': 'IC_1',
    'time marching': {
        'unsteady': {
            'total time': 1.0,
            'time step': 1.0,
            'order': 'second',
        },
        'scheme': {
            'name': 'lu-sgs',
            'stage': 1,
            #'name' : 'runge kutta',
            #'stage': 5,
        },
        'lu-sgs': {
            'Number Of SGS Cycles': 8,
            'Min CFL': 0.1,
            'Max CFL': 5.0,
            'Include Backward Sweep': True,
            'Include Relaxation': True,
            'Jacobian Update Frequency': 1,
            'Jacobian Epsilon': 1.0e-08,
            'CFL growth': 1.05,
            'Use Rusanov Flux For Jacobian': 'true',
            'Finite Difference Jacobian': 'false',
        },

        'multigrid': 10,
        'cfl': 2.5,
        'cfl transport': 2.5 * 0.5,
        'ramp': {'initial': 1.0, 'growth': 1.1},
        'cycles': cycles,
    },

    'equations': 'RANS',

    'RANS': {
        'order': 'euler_second',
        'limiter': 'vanalbada',
        'precondition': 'true',
        'turbulence': {
                   'model': 'sst',
        },
    },

    'material': 'air',
    'air': {
        'gamma': 1.4,
        'gas constant': 287.0,
        'Sutherlands const': 110.4,
        'Prandtl No': 0.72,
        'Turbulent Prandtl No': 0.9,
    },
    'IC_1': {
        'temperature': 310.928,
        'pressure': 101325.0,
        'alpha': alpha,  # User defined variable used for post processing
        'V': {
            'vector': zutil.vector_from_angle(alpha, 0.0),
            'Mach': 0.85,
        },
        'Reynolds No': 5.0e6,
        'Reference Length': 275.8,
        'turbulence intensity': 1.e-4,
        'eddy viscosity ratio': 0.1,
    },
    'BC_1': {
        'ref': 7,
        'type': 'symmetry',
    },
    'BC_2': {
        'ref': 3,
        'type': 'wall',
        'kind': 'noslip',
    },
    'BC_3': {
        'ref': 9,
        'type': 'farfield',
        'condition': 'IC_1',
        'kind': 'riemann',
    },
    'write output': {
        'format': 'vtk',
                  'surface variables': ['V', 'p', 'T', 'rho', 'walldist', 'yplus', 'mach', 'cp', 'eddy', 'pressureforce', 'frictionforce'],
                  'volume variables': ['V', 'p', 'T', 'rho', 'walldist', 'mach', 'cp', 'eddy'],
                  'frequency': 500,
    },
    'report': {
        'frequency': 10,
        'forces': {
            'FR_1': {
                'name': 'wall',
                'zone': [9, 10, 11, 12, 13],
                'transform': my_transform,
                'reference area': 594720.0 * 0.5,  # half model area # half model area # half model area
            },
        },
    },
}

Jinja is a template library and it is certainly not limited to only HTML. But I am not sure why you need to have a template in the first place if all you read is pure python dict from a python file. Although to me it looks like it is a strange way of dealing with confs. Why don't you try to have a conf file in YAML or JSON and that way it will be easy to read, manipulate and write back to S3. [Here](http://docs.aws.amazon.com/lambda/latest/dg/with-s3.html) is a tutorial to access files from / to S3 with AWS lambda. — SRC, Jun 20 '17 at 08:32
@SRC, The application running requires the python file as the format of the config, I'll throw an example into the OP. I suppose I could read the .py, dump the dict into a JSON file, process that and then dump that back into another python file. As for the reading writing to S3, I've got that working already. — Lewiky, Jun 20 '17 at 08:37
Although, this is a terrible and bad solution and I strongly suggest you review your code / architecture so that you do not need to use this solution but if you have a string (or a file containing few lines) and want to execute it as a python code then you can use [eval](https://docs.python.org/3/library/functions.html#eval) or [exec](https://docs.python.org/3/library/functions.html#exec) — SRC, Jun 20 '17 at 08:54
I have no idea how else you can achieve this. However seems to me that [this link](https://stackoverflow.com/questions/32352187/import-python-module-from-external-url) may help. The last answer. — SRC, Jun 20 '17 at 09:20

score 1 · Answer 1 · answered Jun 20 '17 at 09:19

Jinja2 can certainly do it. but if it's worth to do is another question

I modify the your file a little to make it renderable by jinja2

import zutil

alpha = {{ alpha | default(2.13) }}
cycles = {{ cycles | default(100)}}

def my_transform(x,y,z):
    v = [x,y,z]
    v =  zutil.rotate_vector(v,alpha,0.0)
    return {'v1' : v[0], 'v2' : v[1], 'v3' : v[2]}

parameters = { 

 # units for dimensional quantities
'units' : 'SI',

# reference state
'reference' : 'IC_1',

'time marching' : { 
                   'unsteady' : {
                                 'total time' : 1.0,
                                 'time step' : 1.0,
                                 'order' : 'second',
                                },
                   'scheme' : {
                             'name' : 'lu-sgs',
                               'stage': 1,
                               #'name' : 'runge kutta',
                               #'stage': 5,
                               },
                   'lu-sgs' : {
                               'Number Of SGS Cycles' : 8,
                               'Min CFL' : 0.1,
                               'Max CFL' : 5.0,
                               'Include Backward Sweep' : True,
                               'Include Relaxation' : True,
                               'Jacobian Update Frequency' : 1,
                               'Jacobian Epsilon' : 1.0e-08,
                               'CFL growth' : 1.05,
                               'Use Rusanov Flux For Jacobian' : 'true',
                               'Finite Difference Jacobian' : 'false',
                              },

                   'multigrid' : 10,
                   'cfl': 2.5,
                   'cfl transport' : 2.5*0.5,
                   'ramp': { 'initial': 1.0, 'growth': 1.1 },
                   'cycles' : cycles,
                  },

'equations' : 'RANS',

'RANS' : {
               'order' : 'euler_second',
               'limiter' : 'vanalbada',
               'precondition' : 'true',                                          
               'turbulence' : {
                               'model' : 'sst',
                              },
               },

'material' : 'air',
'air' : {
        'gamma' : 1.4,
        'gas constant' : 287.0,
        'Sutherlands const': 110.4,
        'Prandtl No' : 0.72,
        'Turbulent Prandtl No' : 0.9,
        },
'IC_1' : {
          'temperature':310.928,
          'pressure':101325.0,
          'alpha': alpha, # User defined variable used for post processing
          'V': {
                'vector' : zutil.vector_from_angle(alpha,0.0),
                'Mach' : 0.85,
                },
           'Reynolds No' : 5.0e6,
           'Reference Length' : 275.8, 
          'turbulence intensity':1.e-4,
          'eddy viscosity ratio':0.1,
          },
'BC_1' : {
          'ref' : 7,
          'type' : 'symmetry',
         },
'BC_2' : {
          'ref' : 3,
          'type' : 'wall',
          'kind' : 'noslip',
         },
'BC_3' : {
          'ref' : 9,
          'type' : 'farfield',
          'condition' : 'IC_1',
          'kind' : 'riemann',
         },
'write output' : {
                  'format' : 'vtk',
                  'surface variables': ['V','p','T','rho','walldist','yplus','mach','cp','eddy','pressureforce','frictionforce'],
                  'volume variables': ['V','p','T','rho','walldist','mach','cp','eddy'],
                  'frequency' : 500,
                 },      
'report' : {
           'frequency' : 10,
        'forces' : {
            'FR_1' : {
                'name' : 'wall',
                'zone' : [9,10,11,12,13],
                'transform' : my_transform,
                'reference area' : 594720.0*0.5, # half model area # half model area # half model area 
            },
        },
},                      
}

Here is how to render it using jinja2. suppose the path_1 is path of your config file. path_2 is the path of new config file

from jinja2 import Environment, FileSystemLoader

new_config_contex = {'alpha':3, 'cycles': 200}
path, template_filename = os.path.split(path_1)
env = Environment(loader=FileSystemLoader(path))
new_conf_file_content=env.get_template(template_filename).render(new_config_contex)
with open(path_2, "wb") as f:
    f.write(new_conf_file_content)

Thanks for the advice, took it and extended out as in my answer I posted. — Lewiky, Jun 20 '17 at 12:27

score 0 · Answer 2 · answered Jun 20 '17 at 12:27

I found a solution, it's not pretty and as SRC mentioned, it's a bad solution that shouldn't be used in anything real, but it works.

I took advice from milo and converted my reference control.py file into a Jinja template as shown here:

import jinja2
from sys import argv

pyConf = """
import zutil

alpha = {{alpha}}
cycles = {{cycles}}

def my_transform(x,y,z):
    v = [x,y,z]
    v =  zutil.rotate_vector(v,alpha,0.0)
    return {'v1' : v[0], 'v2' : v[1], 'v3' : v[2]}

parameters = {

 # units for dimensional quantities
'units' : 'SI',

# reference state
'reference' : 'IC_1',

'time marching' : {
                   'unsteady' : {
                                 'total time' : 1.0,
                                 'time step' : 1.0,
                                 'order' : 'second',
                                },
                   'scheme' : {
                             'name' : 'lu-sgs',
                               'stage': 1,
                               #'name' : 'runge kutta',
                               #'stage': 5,
                               },
                   'lu-sgs' : {
                               'Number Of SGS Cycles' : 8,
                               'Min CFL' : 0.1,
                               'Max CFL' : 5.0,
                               'Include Backward Sweep' : True,
                               'Include Relaxation' : True,
                               'Jacobian Update Frequency' : 1,
                               'Jacobian Epsilon' : 1.0e-08,
                               'CFL growth' : 1.05,
                               'Use Rusanov Flux For Jacobian' : 'true',
                               'Finite Difference Jacobian' : 'false',
                              },

                   'multigrid' : 10,
                   'cfl': 2.5,
                   'cfl transport' : 2.5*0.5,
                   'ramp': { 'initial': 1.0, 'growth': 1.1 },
                   'cycles' : cycles,
                  },

'equations' : 'RANS',

'RANS' : {
               'order' : 'euler_second',
               'limiter' : 'vanalbada',
               'precondition' : 'true',
               'turbulence' : {
                               'model' : 'sst',
                              },
               },

'material' : 'air',
'air' : {
        'gamma' : 1.4,
        'gas constant' : 287.0,
        'Sutherlands const': 110.4,
        'Prandtl No' : 0.72,
        'Turbulent Prandtl No' : 0.9,
        },
'IC_1' : {
          'temperature':310.928,
          'pressure':101325.0,
          'alpha': alpha, # User defined variable used for post processing
          'V': {
                'vector' : zutil.vector_from_angle(alpha,0.0),
                'Mach' : 0.85,
                },
           'Reynolds No' : 5.0e6,
           'Reference Length' : 275.8,
          'turbulence intensity':1.e-4,
          'eddy viscosity ratio':0.1,
          },
'BC_1' : {
          'ref' : 7,
          'type' : 'symmetry',
         },
'BC_2' : {
          'ref' : 3,
          'type' : 'wall',
          'kind' : 'noslip',
         },
'BC_3' : {
          'ref' : 9,
          'type' : 'farfield',
          'condition' : 'IC_1',
          'kind' : 'riemann',
         },
'write output' : {
                  'format' : 'vtk',
                  'surface variables': ['V','p','T','rho','walldist','yplus','mach','cp','eddy','pressureforce','frictionforce'],
                  'volume variables': ['V','p','T','rho','walldist','mach','cp','eddy'],
                  'frequency' : 500,
                 },
'report' : {
           'frequency' : 10,
        'forces' : {
            'FR_1' : {
                'name' : 'wall',
                'zone' : [9,10,11,12,13],
                'transform' : my_transform,
                'reference area' : 594720.0*0.5, # half model area # half model area # half model area
            },
        },
},
}
"""
template = jinja2.Template(pyConf)
print template.render(alpha = argv[1], cycles = argv[2])

Then I amended my lambda function, to get this template, execute it, which prints the now rendered control file to stdio, where in the lambda function I have redirected the stdio for my exec function, where I can catch it and stream it into an S3 Object, which then gets pushed back to S3.

import boto3
import jinja2 
import sys
import StringIO
import contextlib

@contextlib.contextmanager
def stdoutIO(stdout=None):
    old = sys.stdout
    if stdout is None:
        stdout = StringIO.StringIO()
    sys.stdout = stdout
    yield stdout
    sys.stdout = old

s3 = boto3.resource('s3')

def main(json_input, context):
    client = boto3.client('s3')
    body = client.get_object(Bucket = "cfdworkflow2", Key='Control/control.py').get('Body').read()
    c_body = convertBody(body)
    s3.Object('cfdworkflow2', 'Control/parsedControl.py').put(Body=c_body)

def convertBody(body):
    sys.argv = ['','3','100']
    with stdoutIO() as s:
        exec(body)
    return s.getvalue()

Found the idea of redirecting the stdio here

To extend I'll pass through the stdin to the lambda function as the parameters to the jinja template.

Edit parameters in a python file from another python file

2 Answers2