How to prevent table regeneration in PLY

Question

I am using PLY in a command line application that I package as a Python egg to be installed via pip. Everytime I run my script from the command line, I see the following message:

"Generating LALR tables"

Additionally, parser.out and parsetab.py files are written to the directory from which the script is invoked. Is there some way to ship these files with the application so that it does not regenerate the tables each and every time?

I don't know if it's possible to do this with `ply`. You can change the filename or directory to use, or choose to always regenerate the tables (which may not be an option, depending the size of your grammar). But to ship the parser table generated... don't know. :/ — Elias Dorneles, Oct 01 '12 at 19:10

score 12 · Answer 1 · edited Feb 13 '13 at 22:48

12

use

yacc.yacc(debug=0, write_tables=0)

edited Feb 13 '13 at 22:48

bensiu

24,660
56
77
117

answered Feb 13 '13 at 22:28

eserge

131
1
6

1

Note: you should really use `debug=False, write_tables=False`. The `0` just happens to work but it's more explicit to use `True`/`False` and the documentation also uses `bool` and not integers. – Bakuriu Oct 02 '16 at 12:56

score 3 · Answer 2 · answered Oct 04 '12 at 16:45

You want to use optimized mode, by calling lex as:

lexer = lex.lex(optimize=1)

.

It's worth emphasising (from the same link):

On subsequent executions, lextab.py will simply be imported to build the lexer. This approach substantially improves the startup time of the lexer and it works in Python's optimized mode.

When running in optimized mode, it is important to note that lex disables most error checking. Thus, this is really only recommended if you're sure everything is working correctly and you're ready to start releasing production code.

Since this is production code, this sounds like exactly what you want.

.

In looking into this issue, I came across the miscellaneous Yacc notes:

Since the generation of the LALR tables is relatively expensive, previously generated tables are cached and reused if possible. The decision to regenerate the tables is determined by taking an MD5 checksum of all grammar rules and precedence rules. Only in the event of a mismatch are the tables regenerated.

And looking deeper into the yacc function inside yacc.py, we see that optimise ignores this mismatch in the following snippet:

if optimize or (read_signature == signature):
    try:
        lr.bind_callables(pinfo.pdict)
        parser = LRParser(lr,pinfo.error_func)
        parse = parser.parse
        return parser

where signature is compared to checksum stored in parsetab.py (as _lr_signature).

Is there something you need to do besides setting optimize=1 in the parser and lexer? When I do that, it still regenerates the files and still prints the message. — Michael, Oct 05 '12 at 18:20

score 0 · Accepted Answer · answered Oct 07 '12 at 03:55

What I ultimately wound up doing was turning off optimization. I was going through the PLY 3.4 source and I found this little nugget in the lexer code:

# If in optimize mode, we write the lextab
if lextab and optimize:
    lexobj.writetab(lextab,outputdir)

return lexobj

By changing the code that builds the lexer and parser to:

self.lexer = lex.lex(module=self, optimize=False, debug=False, **kwargs)

and

self.lexer = lex.lex(module=self, optimize=False, debug=False, **kwargs)

I avoided all file write-outs. The debugger writes .out files into the directory and the Python files are the result of the optimize flag.

While this works for the time being, I cannot say I am entirely happy with this approach. Presumably, having some way to keep optimization on and, at the same time, keep the working directory clean would be a superior solution would result in better performance. If someone else has a better methodology, I am more than open to it.

score 0 · Answer 4 · answered Apr 14 '15 at 21:13

This is an old question, but I ran into a similar problem with ply when I tried to use the outputdir yacc keyword argument to place the generated parser tables in specific directories within my project -- it would place them there, but re-generate them every time regardless. I found this patch on github which solved the regeneration issue with no noticeable ill effects. Basically, all it does is modify the read_table method on the yacc class to take an extra parameter -- the outputdir -- and searches the directory there before re-generation. In order to make that work, the sole call site to read_table (in method yacc) also needs to be modified to pass the outputdir keyword argument.

nagylzs · Answer 5 · 2012-10-03T18:04:53.557

-1

Apparently, there are arguments for this in ply.yacc:

def yacc(method='LALR', debug=yaccdebug, module=None, tabmodule=tab_module, start=None, 
     check_recursion=1, optimize=0, write_tables=1, debugfile=debug_file,outputdir='',
     debuglog=None, errorlog = None, picklefile=None):

So, you just pass a different errorlog and debuglog (with a debug() etc. methods that do not print to stdout/stderr). And you specify a fixed outputdir. And that's all you need to do.

UPDATE: I just checked and this is the correct setting:

yacc.yacc(
    debug=False,                         # do not create parser.out
    outputdir=r"c:\temp\aaa" # instruct to place parsetab here
)

Actually you need to use an outputdir that already contains parsetab.py. This will eliminate not just the message but your program will not write out parsetab.py. It will just use it.

edited Oct 03 '12 at 18:04

answered Oct 03 '12 at 14:25

nagylzs

3,954
6
39
70

1

The OP is not asking to simply not showing the message, but not regenerate the temporary parser-tables files all the time. – Elias Dorneles Oct 03 '12 at 17:32
I guess that when you use a fixed dir and you already place the files there, then they won't be regenerated. But I'll try and let you know the results. – nagylzs Oct 03 '12 at 17:56
`debug=False` does indeed suppress the error message, but does not suppress the file generation. When I use it in conjunction with the deployment location (where the files already exist) it still attempts to overwrite them. – Michael Oct 05 '12 at 18:27
If you want to disable file generation completely then overwrite the `LRGeneratedTable.write_table` method (e.g. make it a method that does nothing.) Alternatively, you can use the `picklefile` parameter (passing a file like object) and overwrite `LRGeneratedTable.pickle_table` to do nothing. Unfortunately, the usage of LRGeneratedTable class is hard wired into `yacc.yacc`, so you cannot subclass that. :-( – nagylzs Oct 06 '12 at 16:32

How to prevent table regeneration in PLY

5 Answers5