22

Problem:
I'm running out of RAM while performing the rake assets:precompile task in an automated build. Are there any strategies to do an incremental precompile, or in some other way perform the precompile stage without consuming as much RAM? It appears as though that task consumes around 850 MB more than the baseline for the build.

Context:
I'm trying to get a single Docker container Bitbucket Pipelines version of our automated build. Application stack includes Rails 4.2.7, PostgreSQL 9.3, Java 8, Maven 3.3.9 and JRuby 9.1.2.0. I've tried creating the image based off of Debian Jessie and also off of Alpine Linux, but it doesn't make much difference in the baseline memory.

Toby Murray
  • 521
  • 3
  • 14
  • Try to find out whether you can precompile locally. Capistrano does that. –  Jun 27 '16 at 23:11
  • Is the question too specific? I'm unclear as to why its being down-voted. I can try and change it to be more broadly applicable? – Toby Murray Jun 28 '16 at 14:43
  • I think it is clear enough. Not sure why would it be downvoted. –  Jun 28 '16 at 14:45
  • @gen - the precompile step is *slow*, and we run a lot of builds. A big perk of the CI setup is moving this task off the developer's machine, and we don't have control of the host instance on the CI server. Is that what you were thinking of in terms of "local"? – Toby Murray Jun 28 '16 at 14:54
  • 1
    Yes, I had one project with the similar constrains, and I ended up compiling things during deployment process on my development machine and only uploading the ready files to production. Maybe you can decrease precompile time and resources by moving things to public folder, but it is just my guess, which I never tried myself. –  Jun 28 '16 at 15:04
  • 1
    Does your VPS has SWAP? I had a VPS with 512 MB RAM and faced the same issue. I added a SWAP of 2 or 4 GB and it all worked like a charm. – Jagjot Aug 04 '16 at 04:33
  • @JagjotSingh - good thought, but no, it doesn't support swap space. – Toby Murray Aug 04 '16 at 18:34
  • Which Rails version do you use? Memory and speed footprint differ from version to version. Also using jruby is not this common in my experince. Maybe you can use mri only for asset compilation – slowjack2k Aug 05 '16 at 19:11
  • @slowjack2k Presently we use 4.2.7, hopefully upgrade to 5 once it's matured a little. I think using MRI was one of the first things we tried, we also use Bundler, and it doesn't place nicely with swapping out the Ruby version. Maybe there's some way of extracting the files, precompiling with MRI outside the scope of Bundler and then putting them back in? – Toby Murray Aug 05 '16 at 20:11
  • 1
    @TobyMurray 4.2 should be good enough. I would try to make the gemfile conditional for jruby and mri via `install_if RUBY_VERSION >= '2.0.0' && Bundler.current_ruby.mri? do gem 'rubocop' gem 'codeclimate-test-reporter', :require => false end` and switch ruby version with for instance rvm. Also you can use the environment variable BUNDLE_GEMFILE to use a seperate gemfile for each ruby. A precompile of the assets with --trace can lead to further hints what happens. Maybe you have to many requires insite your assets. Or do you use therubyracer? This uses a lot of memory. – slowjack2k Aug 05 '16 at 20:48
  • @slowjack2k I hadn't seen BUNDLE_GEMFILE before - that looks like it'd address part of the MRI/JRuby puzzle. We don't use therubyracer, we do use therubyrhino though. I'll have to get back to you about the conditional gemfile once I've had a second to play with it. Current executions with --trace don't reveal anything particularly interesting, just 134 assets - mostly small icons as PNGs, then a mix of JS, CSS, etc. – Toby Murray Aug 05 '16 at 21:02
  • 2
    @TobyMurray To many things can cause your issue. I did find a hint that therubyrhino is also slow and needs memory. I did find this command to use node instead of rhino `EXECJS_RUNTIME='Node' JRUBY_OPTS="-J-d32 -X-C" rake assets:precompile` maybe it helps. – slowjack2k Aug 05 '16 at 21:28
  • @slowjack2k - I just had a chance now to try out your suggestion. I installed NodeJS and MRI Ruby 2.3.0, created a separate gemfile that I invoked by setting BUNDLE_GEMFILE, and then ran the precompilation before the build (MRI and NodeJS instead of JRuby and therubyrhino) - great success! If you aggregate your comments as an answer, I'll accept it. – Toby Murray Aug 24 '16 at 18:39
  • I may have celebrated too soon... it looks like the original assets:precompile task isn't recognizing the already precompiled files, and trying to precompile them again (yielding the original issue). To be clear, I left the original task untouched and tried to do this all outside the original build context. Given that they *are* precompiled, this seems like a more solvable problem than the original - reconciling the hashes so they're not precompiled again, or replacing/skipping the original heavy duty precompile. – Toby Murray Aug 24 '16 at 23:03
  • It seems like *the answer* here is to just not use therubyrhino or therubyracer - they have undesirable peak RAM requirements, as noted. Running with Node and ExecJS has dramatically lower RAM requirements. – Toby Murray Aug 25 '16 at 16:09

2 Answers2

4

Short answer

Use NodeJS as the JavaScript interpreter for precompiling (or another JavaScript interpreter characterized by low peak RAM usage).

Longer answer

For context, I'm using NodeJS 4.5.0 as compared to therubyracer v0.12.2 and therubyrhino v2.0.4

Can you increase RAM?

Sounds dumb, but before complicating the build process it may make sense to see if more capable build machines are available, or if swap space is available (although it will likely increase build times).

Can you switch JavaScript interpreters?

High peak RAM utilization seems to be a fundamental characteristic of both therubyrhino (Mozilla's Rhino JavaScript interpreter) and therubyracer (V8 JavaScript interpreter). There does not appear to be an effective way to significantly drop the amount of RAM consumed during the asset precompilation stage. The most viable paths appear to be precompiling the assets outside the build lifecycle and caching them somewhere so they can be fetched instead of precompiled, as suggested by @user4776684. As comments on the question suggest, both Rails version and Ruby version have an impact, but not nearly as much as the JavaScript interpreter.

If all else fails

As @slowjack2k mentioned above, if using Bundler, a parallel configuration can be leveraged to invoke NodeJS only for the precompile task and keep the original build relatively untouched. I didn't look into this as it was easier to switch interpreters, but while I was able to precompile the assets with rake and NodeJS, they didn't appear to be considered precompiled when it came to the rake + therubyrhino invocation so they were re-precompiled. I accomplished this with a programatically set BUNDLE_GEMFILE environment variable which pointed to a completely separate gemfile, which used MRI Ruby and NodeJS instead of JRuby and therubyrhino.

Community
  • 1
  • 1
Toby Murray
  • 521
  • 3
  • 14
0

I have similar issue on cheapest droplet on digitalocean. I created the linux swap partition. Maybe your main host doesn't have swap partition.

MrOnyszko
  • 847
  • 9
  • 13
  • It's a good thought, I [commented above](http://stackoverflow.com/questions/38064461/reduce-memory-consumption-in-rake-assetsprecompile/38961706#comment64921311_38064461) that this host does not have swap. – Toby Murray Aug 15 '16 at 22:38