Short answer
Use NodeJS as the JavaScript interpreter for precompiling (or another JavaScript interpreter characterized by low peak RAM usage).
Longer answer
For context, I'm using NodeJS 4.5.0 as compared to therubyracer v0.12.2 and therubyrhino v2.0.4
Can you increase RAM?
Sounds dumb, but before complicating the build process it may make sense to see if more capable build machines are available, or if swap space is available (although it will likely increase build times).
Can you switch JavaScript interpreters?
High peak RAM utilization seems to be a fundamental characteristic of both therubyrhino (Mozilla's Rhino JavaScript interpreter) and therubyracer (V8 JavaScript interpreter). There does not appear to be an effective way to significantly drop the amount of RAM consumed during the asset precompilation stage. The most viable paths appear to be precompiling the assets outside the build lifecycle and caching them somewhere so they can be fetched instead of precompiled, as suggested by @user4776684. As comments on the question suggest, both Rails version and Ruby version have an impact, but not nearly as much as the JavaScript interpreter.
If all else fails
As @slowjack2k mentioned above, if using Bundler, a parallel configuration can be leveraged to invoke NodeJS only for the precompile task and keep the original build relatively untouched. I didn't look into this as it was easier to switch interpreters, but while I was able to precompile the assets with rake and NodeJS, they didn't appear to be considered precompiled when it came to the rake + therubyrhino invocation so they were re-precompiled. I accomplished this with a programatically set BUNDLE_GEMFILE
environment variable which pointed to a completely separate gemfile, which used MRI Ruby and NodeJS instead of JRuby and therubyrhino.