27

Are there any projects that used node.js and closure-compiler (CC for short) together?

The official CC recommendation is to compile all code for an application together, but when I compile some simple node.js code which contains a require("./MyLib.js"), that line is put directly into the output, but it doesn't make any sense in that context.

I see a few options:

  1. Code the entire application as a single file. This solves the problem by avoiding it, but is bad for maintenance.
  2. Assume that all files will be concatenated before execution. Again this avoids the problem, but makes it harder to implement a un-compiled debug mode.
  3. I'd like to get CC to "understand" the node.js require() function, but that probably can't be done without editing the compiler itself, can it?
kryger
  • 12,906
  • 8
  • 44
  • 65
bukzor
  • 37,539
  • 11
  • 77
  • 111
  • `require` means that you're using RequireJS or some other form of AMD loader. Some AMD-type systems contain a tool for you to "flatten" all the dependencies and create a single file with all the necessary code. That's what you need to do to use Closure -- have one file containing all the code. – Stephen Chung Nov 28 '11 at 15:33
  • @StephenChung I think he's trying to run closure on node.js code not on browser side code. – Raynos Nov 28 '11 at 16:00
  • I know that Michael Bolin has done some work to get Closure Library working with Node.js. I don't know if this extended to using the Compiler as well. – John Nov 28 '11 at 16:25
  • 2
    @StephenChung: Advanced Micro Devices? – bukzor Nov 29 '11 at 19:13
  • @John: Link please? When I search "michael bolin node.js", this discussion is the second result. – bukzor Nov 29 '11 at 19:21
  • @bukzor: https://github.com/amdjs/amdjs-api/wiki/AMD , but it's not relevant to your question I think. – Nickolay Nov 30 '11 at 19:18
  • and the other comment probably references this: https://github.com/bolinfest/node-google-closure-latitude-experiment – Nickolay Nov 30 '11 at 19:21
  • I asked Michael to respond here, which he has, below. – John Dec 02 '11 at 16:29
  • Yeah, require is node.js's require in this case, not AMD's. – aredridel Jan 31 '13 at 18:50

5 Answers5

51

I have been using the Closure Compiler with Node for a project I haven't released yet. It has taken a bit of tooling, but it has helped catch many errors and has a pretty short edit-restart-test cycle.

First, I use plovr (which is a project that I created and maintain) in order to use the Closure Compiler, Library, and Templates together. I write my Node code in the style of the Closure Library, so each file defines its own class or collection of utilities (like goog.array).

The next step is to create a bunch of externs files for the Node functions you want to use. I published some of these publicly at:

https://github.com/bolinfest/node-google-closure-latitude-experiment/tree/master/externs/node/v0.4.8

Though ultimately, I think that this should be a more community driven thing because there are a lot of functions to document. (It's also annoying because some Node functions have optional middle arguments rather than last arguments, making the type annotations complicated.) I haven't started this movement myself because it's possible that we could do some work with the Closure Complier to make this less awkward (see below).

Say you have created the externs file for the Node namespace http. In my system, I have decided that anytime I need http, I will include it via:

var http = require('http');

Though I do not include that require() call in my code. Instead, I use the output-wrapper feature of the Closure Compiler the prepend all of the require()s at the start of the file, which when declared in plovr, in my current project looks like this:

"output-wrapper": [
  // Because the server code depends on goog.net.Cookies, which references the
  // global variable "document" when instantiating goog.net.cookies, we must
  // supply a dummy global object for document.
  "var document = {};\n",

  "var bee = require('beeline');\n",
  "var crypto = require('crypto');\n",
  "var fs = require('fs');\n",
  "var http = require('http');\n",
  "var https = require('https');\n",
  "var mongodb = require('mongodb');\n",
  "var nodePath = require('path');\n",
  "var nodeUrl = require('url');\n",
  "var querystring = require('querystring');\n",
  "var SocketIo = require('socket.io');\n",
  "%output%"
],

In this way, my library code never calls Node's require(), but the Compiler tolerates the uses of things like http in my code because the Compiler recognizes them as externs. As they are not true externs, they have to be prepended as I described.

Ultimately, after talking about this on the discussion list, I think the better solution is to have a new type annotation for namespaces that would look something like:

goog.scope(function() {

    /** @type {~NodeHttpNamesapce} */
    var http = require('http');

    // Use http throughout.

});

In this scenario, an externs file would define the NodeHttpNamespace such that the Closure Compiler would be able to typecheck properties on it using the externs file. The difference here is that you could name the return value of require() whatever you wanted because the type of http would be this special namespace type. (Identifying a "jQuery namespace" for $ is a similar issue.) This approach would eliminate the need to name your local variables for Node namespaces consistently, and would eliminate the need for that giant output-wrapper in the plovr config.

But that was a digression...once I have things set up as described above, I have a shell script that:

  1. Uses plovr to build everything in RAW mode.
  2. Runs node on the file generated by plovr.

Using RAW mode results in a large concatenation of all the files (though it also takes care of translating Soy templates and even CoffeeScript to JavaScript). Admittedly, this makes debugging a pain because the line numbers are nonsense, but has been working well enough for me so far. All of the checks performed by the Closure Compiler have made it worth it.

bolinfest
  • 3,710
  • 2
  • 27
  • 40
  • Thanks for the detailed answer! Could you link your referenced discussion? – bukzor Dec 03 '11 at 07:24
  • 1
    Updated my answer to link to the discussion at https://groups.google.com/forum/#!searchin/closure-compiler-discuss/node/closure-compiler-discuss/jws6kzCp4LI/gbIamVk4gAcJ – bolinfest Dec 04 '11 at 16:26
  • Is this going to get simpler in the future? It seems like a lot of hoops. – bukzor Dec 21 '12 at 05:58
  • @bukzor: maybe when Google Closure Compiler (GCC) supports node.js specifically. The problem is in the node.js core modules, which are confusing to GCC. The other `require()`'s are supported by `--common_js_modules` and `--transform_amd_modules`. But I doubt it will be supported any time soon because the performance gain is probably minimal. V8 performs a lot of the same optimizations that GCC does, like dead code removal and inlining. Maybe the memory footprint will be a bit smaller, but that will not make a substantial difference. – Blaise Jan 12 '13 at 11:10
  • I managed to get node.js code compiling working using Google Closure's REST API. I did the inlining of non-core `require()`'s myself before compiling, sending the API one big flattened file with Node.js core modules removed and externed. Then after compiling, I readd the Node.js core requires. --- See https://github.com/blaise-io/xssnake/blob/master/build/server.js and https://github.com/blaise-io/xssnake/blob/master/build/lib/server_compile.js – Blaise Jan 12 '13 at 20:45
  • @Blaise: That seems very interesting, but I don't truly understand your code. I'm a Python guy. If you'd make a new answer that shows how to do this in a minimal project (the posted `xssnake` project seems large), that'd be way cool. – bukzor Jan 13 '13 at 19:29
6

The svn HEAD of closure compiler seems to have support for AMD

Jauco
  • 1,320
  • 13
  • 23
3

I replaced my old approach with a way simpler approach:

New approach

  • No require() calls for my own app code, only for Node modules
  • I need to concatenate server code to a single file before I can run or compile it
  • Concatenating and compiling is done using a simple grunt script

Funny thing is that I didn't even had to add an extern for the require() calls. The Google Closure compiler understands that automagically. I did have to add externs for nodejs modules that I use.

Old approach

As requested by OP, I will elaborated on my way of compiling node.js code with Google Closure Compiler.

I was inspired by the way bolinfest solved the problem and my solution uses the same principle. The difference is that I made one node.js script that does everything, including inlining modules (bolinfest's solution lets GCC take care of that). This makes it more automated, but also more fragile.

I just added code comments to every step I take to compile server code. See this commit: https://github.com/blaise-io/xssnake/commit/da52219567b3941f13b8d94e36f743b0cbef44a3

To summarize:

  1. I start with my main module, the JS file that I pass to Node when I want to run it.
    In my case, this file is start.js.
  2. In this file, using a regular expression, I detect all require() calls, including the assignment part.
    In start.js, this matches one require call: var Server = require('./lib/server.js');
  3. I retrieve the path where the file exists based on the file name, fetch its contents as a string, and remove module.exports assignments within the contents.
  4. Then I replace the require call from step 2 with the contents from step 3. Unless it is a core node.js module, then I add it to a list of core modules that I save for later.
  5. Step 3 will probably contain more require() call, so I repeat step 3 and 4 recursively until all require() calls are gone and I'm left with one huge string containing all code.
  6. If all recursion has completed, I compile the code using the REST API.
    You could also use the offline compiler.
    I have externs for every core node.js module. This tool is useful for generating externs.
  7. I preprend the removed core.js module require calls to the compiled code.

Pre-Compiled code.
All require calls are removed. All my code is flattened.
http://pastebin.com/eC2rVMiN

Post-Compiled code.
Node.js core require calls have been prepended manually.
http://pastebin.com/uB8CaejN


Why you should not do it this way:

  1. It uses regular expressions (not a parser or tokenizer) for detecting require calls, inlining and removing module.exports. This is fragile, as it does not cover all syntax variations.
  2. When inlining, all module code is added to the global namespace. This is against the principles of Node.js, where every file has its own namespace, and this will cause errors if you have two different modules with the same global variables.
  3. It does not improve the speed of your code that much, since V8 also performs a lot of code optimizations like inlining and dead code removal.

Why you should:

  1. Because it does work when you have consistent code.
  2. It will detect errors in your server code when you enable verbose warnings.

Blaise
  • 13,139
  • 9
  • 69
  • 97
  • I know smjs has a [parser interface](https://developer.mozilla.org/en-US/docs/SpiderMonkey/Parser_API). Does v8 expose anything similar? – bukzor Jan 15 '13 at 20:07
  • I couldn't find any documentation on v8 exposing a parser. However, it is possible to use a JS parser written in JS, like [Esprima](https://npmjs.org/package/esprima). [It outputs this](http://pastebin.com/vC4e1Rec) for [a trivial module](http://pastebin.com/XAKfvC5G). With code chopped up nicely, you could do what I do with regular expressions in a rock-solid way. But it would be a lot of work to make the modifications and then putting the code back together. [This output for one of my real modules probably illustrates that](http://pastebin.com/j32SKq05). – Blaise Jan 16 '13 at 13:46
  • It seems like esprima is an implementation of the api I linked, which doesn't really give consideration to reversibility. In a project of my own, I've done [a second pass on the smjs ast to make it reversible](https://github.com/bukzor/RefactorLib/blob/master/refactorlib/javascript/parse.py). Maybe it's helpful? Probably not =/ – bukzor Jan 18 '13 at 17:09
  • I didn't notice this last time: Esprima rewrites source using [an external library called Escodegen](https://github.com/Constellation/escodegen). Escodegen is apparently compatible with Esprima, as you can see in the [code of their minify example](http://esprima.org/demo/minify.js). --- For me, a Python solution is not compatible with my brain as I'm OCD about doing everything in JS. However, I might modify my hacky library to use both Esprima and Escodegen to create a solid module that compiles *all* node.js projects using Closure Compiler, not just mine. – Blaise Jan 19 '13 at 11:26
  • That seems like the right route. I can't promised I'd "use" it, but I very much like the idea. – bukzor Jan 19 '13 at 18:47
  • Premise: I'd like to use the closure compiler, because I think that the compiled JSDocs are extremely useful. Question: does it make sense to use the closure compiler, given that most of the codebase (most NPMs, I presume) do not have type annotations? (Writing externs for each thing you'd like to use would be too painful for me.) Thanks for the answer! – Chris Sep 18 '15 at 06:43
  • Writing externs is quite easy when you use a tool like http://www.dotnetwise.com/Code/Externs/ – Blaise Sep 18 '15 at 10:02
3

Closure Library on Node.js in 60 seconds.

It's supported, check https://code.google.com/p/closure-library/wiki/NodeJS.

Daniel Steigerwald
  • 1,075
  • 1
  • 9
  • 19
-29

Option 4: Don't use closure compiler.

People in the node community don't tend to use it. You don't need to minify node.js source code, that's silly.

There's simply no good use for minification.

As for the performance benefits of closure, I personally doubt it actually makes your programs faster.

And of course there's a cost, debugging compiled JavaScript is a nightmare

Raynos
  • 166,823
  • 56
  • 351
  • 396
  • 7
    FYI: I wanted to use the optimization and static analysis aspects, rather than just minification. – bukzor Nov 28 '11 at 06:33
  • 25
    @Raynos, I beg to disagree. Closure is not only for minification. There are definitely performance benefits. The OP asks how it can be done, not whether he should do it in the first place. Nobody in the node community uses it doesn't mean that it shouldn't be used. – Stephen Chung Nov 28 '11 at 15:36
  • 2
    @StephenChung I fixed it to be more correct. I don't think there are performance benefits that are worth while. It's sensible to use CC on browser code but not on server side code. – Raynos Nov 28 '11 at 15:58
  • @Raynos, I see your point. However, my view is that some optimizations are valid mostly for browsers (e.g. smaller download), but many will have performance benefits that may add up when scaling to huge number of requests. E.g. dead code removal *may* reduce memory footprint, flattening of namespaces may gain some speed-up in code with deep namespace trees, in-lining may reduce redundant function calls, and prototype virtualization may generate some speed-up when using classes (but probably no when using V8). I agree the benefits are small, though, but may add up. – Stephen Chung Nov 29 '11 at 02:13
  • 3
    @StephenChung The speed up benefits are minor and insignificant. If performance matters rewrite it in C++ instead of running your code through CC. – Raynos Nov 29 '11 at 02:33
  • @Raynos, agreed. Probably only very minor performance increases on a server-class machine. – Stephen Chung Nov 29 '11 at 03:58
  • 1
    @StephenChung, Raynos: I was in fact most interested in gaining the benefits of strong type checking and static error analysis. The optimizations are a nice side-benefit though. See: http://code.google.com/closure/compiler/docs/api-tutorial3.html#better – bukzor Nov 29 '11 at 19:17
  • 13
    I'd also like to point out that "option 4" is not a solution to the problem. "No" can't be an answer to "How?" – bukzor Nov 29 '11 at 19:18
  • @bukzor I don't see "how" CC does type checking or static analysis. Have you tried jshint? – Raynos Nov 29 '11 at 19:25
  • jshint is ok, but occupies a different space. I had planned to use both. – bukzor Nov 29 '11 at 20:02
  • 1
    @Raynos, well it does... to the extent that it will refuse to compile a JavaScript file with type errors in it, not just a warning. So for example, if you mark a function as accepting a string, and you pass in a variable that is not marked to be exclusively holding a string, then it will choke unless you turn off type checking. Sometimes this strictness is frustrating, but it more often than not forces good coding. – Stephen Chung Nov 30 '11 at 04:44
  • 1
    @Raynos, and static analysis is always done before doing optimizations when using Advanced mode. – Stephen Chung Nov 30 '11 at 04:44
  • 4
    @StephenChung that does actually sound rather useful – Raynos Nov 30 '11 at 04:59
  • @Raynos, well I think it is useful. However, it can be !@#$%#@#! frustrating sometimes when it gets too strict... :-) – Stephen Chung Dec 01 '11 at 01:50
  • I think the only performance increase would be in startup since v8 compiles code into machine code beforehand. – Farid Nouri Neshat Dec 02 '11 at 07:46
  • This does not constitute a helpful answer... Minifying is userful – h4unt3r Apr 27 '13 at 03:59
  • @bukzor if it's not a solution to the problem, why did you accept it? – Tomas Jan 08 '14 at 15:35
  • @Tomas: That's a good question. I can't say. It must have been a miss-click. fixed. – bukzor Jan 08 '14 at 19:13
  • @h4unt3r: Minifying server-side code is far less useful. The main benefit client-side code sees from it (lower bandwidth) is effectively absent when the code is local and thus doesn't have to be downloaded. – cHao Feb 12 '14 at 03:33
  • @cHao There are still benefits as mentioned for removing dead code and inlining function calls, its not just minifying (thats the end result not the whole picture) – h4unt3r Feb 12 '14 at 03:49
  • @h4unt3r: You think v8 doesn't already do a bunch of that (or make it unnecessary altogether) when it translates to native code? – cHao Feb 12 '14 at 04:11
  • @bukzor if someone is asking for rope to hang themselves then "no" is a good answer. – Zephyr was a Friend of Mine Mar 21 '14 at 10:24