lightweight javascript to javascript parser

Question

How would I go about writing a lightweight javascript to javascript parser. Something simple that can convert some snippets of code.

I would like to basically make the internal scope objects in functions public.

So something like this

var outer = 42;
window.addEventListener('load', function() {
   var inner = 42;
   function magic() {
       var in_magic = inner + outer;
       console.log(in_magic);
   }
   magic();
}, false);

Would compile to

__Scope__.set('outer', 42);
__Scope__.set('console', console);
window.addEventListener('load', constructScopeWrapper(__Scope__, function(__Scope__) {
    __Scope__.set('inner', 42);
    __Scope__.set('magic',constructScopeWrapper(__Scope__, function _magic(__Scope__) {
        __Scope__.set('in_magic', __Scope__.get('inner') + __Scope__.get('outer'));
        __Scope__.get('console').log(__Scope__.get('in_magic'));
    }));
    __Scope__.get('magic')();
}), false);

Demonstation Example

Motivation behind this is to serialize the state of functions and closures and keep them synchronized across different machines (client, server, multiple servers). For this I would need a representation of [[Scope]]

Questions:

1. Can I do this kind of compiler without writing a full JavaScript -> (slightly different) JavaScript compiler?
2. How would I go about writing such a compiler?
3. Can I re-use existing js -> js compilers?

Answer 1

I don't think your task is easy or short given that you want to access and restore all the program state. One of the issues is that you might have to capture the program state at any moment during a computation, right? That means the example as shown isn't quite right; that captures state sort of before execution of that code (except that you've precomputed the sum that initializes magic, and that won't happen before the code runs for the original JavaScript). I assume you might want to capture the state at any instant during execution.

The way you've stated your problem, is you want a JavaScript parser in JavaScript. I assume you are imagining that your existing JavaScript code J, includes such a JavaScript parser and whatever else is necessary to generate your resulting code G, and that when J starts up it feeds copies of itself to G, manufacturing the serialization code S and somehow loading that up. (I think G is pretty big and hoary if it can handle all of Javascript) So your JavaScript image contains J, big G, S and does an expensive operation (feed J to G) when it starts up.

What I think might serve you better is a tool G that processes your original JavaScript code J offline, and generates program state/closure serialization code S (to save and restore that state) that can be added to/replace J for execution. J+S are sent to the client, who never sees G or its execution. This decouples the generation of S from the runtime execution of J, saving on client execution time and space.

In this case, you want a tool that will make generation of such code S easiest. A pure JavaScript parser is a start but isn't likely enough; you'll need symbol table support to know which function code is connected a function call F(...), and which variable definition in which scope corresponds to assignments or accesses to a variable V. You may need to actually modify your original code J to insert points of access where the program state can be captured. You may need flow analysis to find out where some values went. Insisting all of this in JavaScript narrows your range of solutions.

For these tasks, you will likely find a program transformation tool useful. Such tools contain parsers for the langauge of interest, build ASTs representing the program, enable the construction of identifier-to-definition maps ("symbol tables"), can carry out modifications to the ASTs representing insertion of access points, or synthesis of ASTs representing your demonstration example, and then regenerate valid JavaScript code containing the modified J and the additions S. Of all the program transformation systems that I know about (which includes all the ones at the Wikipedia site), none are implemented in JavaScript.

Our DMS Software Reengineering Toolkit is such a program transformation system offering all the features I just described. (Yes, its big and hoary; it has to be to handle the complexities of real computer languages). It has a JavaScript front end that contains a complete JavaScript parser to ASTs, and the machinery to regenerate JavaScript code from modified or synthesized ASTs. (Also big and hoary; good thing that hoary + hoary is still just hoary). Should it be useful, DMS also provides support for building control and dataflow analysis.

Answer 2

I'd try to look for an existing parser to modify. Perhaps you could adapt JSLint/JSHint?

Answer 3

If you want something with a simple interface, you could try node-burrito: https://github.com/substack/node-burrito

It generates an AST using the uglify-js parser and then recursively walks the nodes. All you have to do is give a single callback which tests each node. You can alter the ones you need to change, and it outputs the resulting code.

Answer 4

There is a problem with the rewriting above, you're not hoisting the initialization of magic to the top of the scope.

There's a number of projects out there that parse JavaScript.

1. Crock's Pratt parser which works well on JavaScript that fits within "The good parts" and less well on other JS.
2. The es-lab parser based on ometa which handles the full grammar including a lot of corner cases that Crock's parser misses. It may not perform as well as Crock's.
3. narcissus parser and evaluator. I don't have much experience with this.

There are also a number of high-quality lexers for JavaScript that let you manipulate JS at the token level. This can be tougher than it sounds though since JavaScript is not lexically regular, and predicting semicolon insertion is difficult without a full parse.

My es5-lexer is a carefully constructed and efficient lexer for EcmaScript 5 that provides the ability to tokenize JavaScript. It is heuristic where JavaScript's grammar is not lexically regular but the heuristic is very good and it provides a means to transform a token stream so that an interpreter is guaranteed to interpret it the way the lexer interpreted the tokens so if you don't trust your input, you can still be sure that the interpretation underlying the security transformations is sound even if not correct according to the spec for some bizarre inputs.

Answer 5

Your problem seams to be in same family of problems as what is solved with the JS Opfuscators and JS Compressors -- they as well as you need to be able to parse and reformat the JS to an equivalent script;

There was a good discussion on obfuscators here and the possible solution to your problem could be to leverage the parse and generator part from one of the FOSS versions.

One callout, your example code does not take into account the scopes of the variables you want to set/get and that will eventually become a problem that you will have to solve.

Addition

Given the scope problem for closure defined functions, you are probably unlikely to be able to solve this problem as a static parsing problem, as the scope variables outside the closure will have to be imported/exported to resolve/save and re-instantiate scope. Hence you may need to dig into the evaluation engine itself, and perhaps get the V8 engine and make a hack to the interpreter itself -- that is assuming that you do not need this to be generic cross all script engines and that you can tie it down to a single implementation which you control.