Most efficient way to modify variables within a custom grammar?

Question

I'm working with a propriety driving simulator which generates "scenario" files in a customised version of Scilab. I am provided a single 11,000 line long "master" file, and from this I need to replace certain values to generate n versions of the scenario.

A minimal example of the syntax for a single parent TASK would be something like this:

TYPEOF TASK (57)
{
    LABEL="Dot 3a"/*replace with name for name in list */
    TASK_KIND="0"

    TYPEOF VARIABLE (53)
    {
        LABEL="Time1"
        TYPE="FLOAT"
        VALUE="14.000000" /* replace with random.integer() */
        INTERACTIVE="VOID"

    TYPEOF VARIABLE (54)
    {
        LABEL="X_pos1"
        TYPE="FLOAT"
        VALUE="23.600000" 
        INTERACTIVE="VOID"


    TYPEOF TASK (58)
    {
        LABEL="Task: ISI"
        TASK_KIND="0"

        TYPEOF RULE (115)
        {
            LABEL="Rule: Go to subtask after Time1 seconds"

            TYPEOF CONDITION (SUPERIOR)
            {
                IS_EXPANDED="1"
                MODIFIER="BECOMES_TRUE"

                TYPEOF PARAMETER (OPERAND_1)
                {
                    KIND="FUNCTION"

                    TYPEOF FUNCTION (GET_TASK_CLOCK)
                    {
                    }
                    OWNER_FILE=""
                }

                TYPEOF PARAMETER (OPERAND_2)
                {
                    KIND="VARIABLE"
                    VALUE="53"
                    OWNER_FILE=""
                }
            }

            TYPEOF ACTION (GOTO_TASK)
            {
                IS_EXPANDED="1"

                TYPEOF PARAMETER (TASK_NUMBER)
                {
                    KIND="ENUM"
                    VALUE="GOTO_NEXT_TASK"
                    OWNER_FILE=""
                }
            }
        }
    }

I need to replace certain values in this script with standard input. For instance, have a list of names which will replace the value of LABEL under a parent level TASK; and have to replace VALUE for first parent VARIABLE with a random number between 6 and 16.

My first solution was Python REGEX based, something like follows (but for every value I seek to change):

for row in scenarioInput:
    parenttaskmatch = re.search("^\t\tTYPEOF TASK",row)
    if parenttaskmatch:
        replacementrow = re.sub(r"([0-9]{1,3})",repl,row)

It was suggested to me that I could write a custom grammar with something like Parsimonious and then regenerate the output with Mustache.

from parsimonious.grammar import Grammar
grammar = Grammar(r"""
    any = task / data
    task = "TYPEOF " key " (" number ")" newline open_curly any+ close_curly
    data = key "=" quote text quote newline
    open_curly = "{" newline
    close_curly = "}" newline
    key = ~"[A-Z 0-9_]*"
    text = ~"[A-Z0-9 ]*"i
    number = ~"[0-9]*"
    newline = "\n"
    space = " "
    quote = "\""
""")

text = open('example_driving_rule.sci').read()
grammar.parse(text)
# Note doesn't work

As you can see, this is not an efficient solution to the problem either. What do you guys think is a better solution?

You probably do want a real parser. Regexs are hard to guarantee they will handle things right if you have nasty things like quoted strings or something. I've never heard of Parsimonious before but it looks like a good place to start! Why doesn't it work though? — Jason S, Aug 18 '16 at 13:37

score 1 · Answer 1 · answered Aug 18 '16 at 13:34

May be you can transform your file to a Scilab Script which generate a file with the new values.

The transformation is quite simple First in Scilab (to be done once)

 T=mgetl("Task_file");mputl(sci2exp(T),"Task_file.sce")

For each experiment, with a text editor modify the generated script to replace the default values by the expected one (may be by reading these values from a file, or ....)

See the example below Time1 value is generated by grand, X_pos1 is read from Scilab console

 T=["TYPEOF TASK (57)"
 "{"
 "    LABEL=""Dot 3a""/*replace with name for name in list */"
 "    TASK_KIND=""0"""
 ""
 "    TYPEOF VARIABLE (53)"
 "    {"
 "        LABEL=""Time1"""
 "        TYPE=""FLOAT"""
 "        VALUE="""+string(grand(1,1,"uin",6,16)+"""" /* replace with random.integer() */"
 "        INTERACTIVE=""VOID"""
 ""
 "    TYPEOF VARIABLE (54)"
 "    {"
 "        LABEL=""X_pos1"""
 "        TYPE=""FLOAT"""
 "        VALUE=""""+string(input("X_pos1")+""""
 "        INTERACTIVE=""VOID"""
 ""
 ""
 "    TYPEOF TASK (58)"
 "    {"
 "        LABEL=""Task: ISI"""
 "        TASK_KIND=""0"""
 ""
 "        TYPEOF RULE (115)"
 "        {"
 "            LABEL=""Rule: Go to subtask after Time1 seconds"""
 ""
 "            TYPEOF CONDITION (SUPERIOR)"
 "            {"
 "                IS_EXPANDED=""1"""
 "                MODIFIER=""BECOMES_TRUE"""
 ""
 "                TYPEOF PARAMETER (OPERAND_1)"
 "                {"
 "                    KIND=""FUNCTION"""
 ""
 "                    TYPEOF FUNCTION (GET_TASK_CLOCK)"
 "                    {"
 "                    }"
 "                    OWNER_FILE="""""
 "                }"
 ""
 "                TYPEOF PARAMETER (OPERAND_2)"
 "                {"
 "                    KIND=""VARIABLE"""
 "                    VALUE=""53"""
 "                    OWNER_FILE="""""
 "                }"
 "            }"
 ""
 "            TYPEOF ACTION (GOTO_TASK)"
 "            {"
 "                IS_EXPANDED=""1"""
 ""
 "                TYPEOF PARAMETER (TASK_NUMBER)"
 "                {"
 "                    KIND=""ENUM"""
 "                    VALUE=""GOTO_NEXT_TASK"""
 "                    OWNER_FILE="""""
 "                }"
 "            }"
 "        }"
 "    }"];
 muptl(T,"Task")

Is it possible to get a solution like this without using scilab? I.e, in bash you could do $1 to get the first variable. I guess I could replace them all with a unique ID like $$LABEL$$, but that returns to the regex example. — Doctor David Anderson, Aug 19 '16 at 09:16

Most efficient way to modify variables within a custom grammar?

1 Answers1