Can Selenium and Chrome buildpack installs be cached on HerokuCI?

Question

I've just started using HerokuCI. I have a few Selenium tests so I've added buildpacks in my app.json.

{
  "buildpacks": [
    { "url": "heroku/ruby" }
  ],
  "environments": {
    "test": {
      "addons": ["heroku-postgresql:in-dyno", "heroku-redis:in-dyno"],
      "buildpacks": [
        { "url": "heroku/ruby" },
        { "url": "https://github.com/notvad/heroku-buildpack-selenium" },
        { "url": "https://github.com/heroku/heroku-buildpack-chromedriver" },
        { "url": "https://github.com/heroku/heroku-buildpack-google-chrome" }
      ],
      "env": {
        "REDIS_PROVIDER": "REDIS_URL",
        "RAILS_ENV": "test",
        "RACK_ENV": "test",
        "MALLOC_ARENA_MAX": 2,
        "LANG": "en_US.UTF-8",
        "TEST_API": 1
      },
      "scripts": {
        "test-setup": "./bin/setup-cc-test-reporter",
        "test": "./bin/test-with-cc-test-reporter"
      }
    }
  }
}

These keep getting rebuilt along with all their dependencies.

-----> Selenium app detected
bash: /tmp/buildpacks/b5ae60a477f2ba582d00a4ba23348059d40dea69cea6d982947f7c6afa72d2b8d69252449d43cbcf97a23341fd57300a3fef3224ace95482e1c0deaf8a844e4d/export: No such file or directory
bash: /tmp/buildpacks/b7af5642714be4eddaa5f35e2b4c36176b839b4abcd9bfe57ee71c358d71152b4fd2cf925c5b6e6816adee359c4f0f966b663a7f8649b0729509d510091abc07/export: No such file or directory
-----> Updating apt caches
       Hit:1 http://apt.postgresql.org/pub/repos/apt bionic-pgdg InRelease
       Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
       Hit:3 http://archive.ubuntu.com/ubuntu bionic-security InRelease
       Hit:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
       Reading package lists...
-----> Fetching .debs for xserver-xorg-core libgl1-mesa-dri xvfb libglide3 xfs unzip
       Reading package lists...
       Building dependency tree...
       Package xfs is not available, but is referred to by another package.
       This may mean that the package is missing, has been obsoleted, or
       is only available from another source

E: Package 'xfs' has no installation candidate
-----> Installing adwaita-icon-theme_3.28.0-1ubuntu1_all.deb
-----> Installing at-spi2-core_2.28.0-1_amd64.deb

...and so on...

-----> Downloading and installing Selenium
-----> Writing profile script
-----> chromedriver app detected
bash: /tmp/buildpacks/b5ae60a477f2ba582d00a4ba23348059d40dea69cea6d982947f7c6afa72d2b8d69252449d43cbcf97a23341fd57300a3fef3224ace95482e1c0deaf8a844e4d/export: No such file or directory
bash: /tmp/buildpacks/b7af5642714be4eddaa5f35e2b4c36176b839b4abcd9bfe57ee71c358d71152b4fd2cf925c5b6e6816adee359c4f0f966b663a7f8649b0729509d510091abc07/export: No such file or directory
bash: /tmp/buildpacks/f55d0db8af2b8c70f2c0e028ef5430955852d9e4f07a06e333e54eb5264a81b9d1e77d1cc02851b4c7da30f6da47b13959e18a24c630212b899920d05f32a566/export: No such file or directory
-----> Looking up latest chromedriver version...
-----> Downloading chromedriver v2.43...
Archive:  /tmp/chromedriver.zip
  inflating: /app/.chromedriver/bin/chromedriver  
-----> Creating chromedriver export scripts...
-----> Google Chrome app detected
bash: /tmp/buildpacks/b5ae60a477f2ba582d00a4ba23348059d40dea69cea6d982947f7c6afa72d2b8d69252449d43cbcf97a23341fd57300a3fef3224ace95482e1c0deaf8a844e4d/export: No such file or directory
bash: /tmp/buildpacks/b7af5642714be4eddaa5f35e2b4c36176b839b4abcd9bfe57ee71c358d71152b4fd2cf925c5b6e6816adee359c4f0f966b663a7f8649b0729509d510091abc07/export: No such file or directory
bash: /tmp/buildpacks/f55d0db8af2b8c70f2c0e028ef5430955852d9e4f07a06e333e54eb5264a81b9d1e77d1cc02851b4c7da30f6da47b13959e18a24c630212b899920d05f32a566/export: No such file or directory
bash: /tmp/buildpacks/27353aa3b5edb2f8361186206e863f5d481f02805d02f0efdec439f3d30349a4fbdad2951af33a0492023fb0d90d26dafda6fe76f1b2d34f68564a98545f7022/export: No such file or directory
-----> Installing Google Chrome from the stable channel.
-----> Updating apt caches
       Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
       Hit:2 http://apt.postgresql.org/pub/repos/apt bionic-pgdg InRelease
       Hit:3 http://archive.ubuntu.com/ubuntu bionic-security InRelease
       Hit:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
       Reading package lists...
-----> Fetching .debs for gconf-service

...and so on...

This is taking up most of the test run time.

Can these installs be cached?

@Bsquare Sorry, I'm not sure I understand what you're asking. — Schwern, Nov 29 '18 at 16:26
I mean, what is the command you launch to get this behaviour? — Bsquare ℬℬ, Nov 29 '18 at 16:27
@Bsquare I don't have direct control. It's launched as a Github PR hook provided by Heroku. I don't control provisioning the CI dyno directly. Instead it's by specifying [buildpacks in a configuration file](https://devcenter.heroku.com/articles/app-json-schema#buildpacks). See https://devcenter.heroku.com/articles/heroku-ci#configuring-your-test-environment — Schwern, Nov 29 '18 at 16:41
@Bsquare Thanks. It might be worth looking into the individual build packs themselves. https://github.com/notvad/heroku-buildpack-selenium for example. And how buildpacks are made, particularly [the caching option](https://devcenter.heroku.com/articles/buildpack-api#caching). Thank you for your help. — Schwern, Nov 29 '18 at 17:21
What is your github project? I'd like to check the corresponding hook. — Bsquare ℬℬ, Dec 05 '18 at 08:36
In addition, can you provide the whole contents of app.json? — Bsquare ℬℬ, Dec 05 '18 at 08:43
@Bsquare I've added the whole app.json. I'll comment on your answer. — Schwern, Dec 05 '18 at 19:34
@Bsquare Sorry, I can't show you the project. It's private. The hook is https://kolkrabbi.heroku.com/hooks/github, "application/json", and "Send me everything". It was set up by Heroku. I suspect all it's doing is informing Heroku there's a new commit to pull and deploy. — Schwern, Dec 05 '18 at 19:47
I think I don't have access to your link, because I got the following json: **{"id":"not_found","message":"Not found."}** — Bsquare ℬℬ, Dec 06 '18 at 08:46
@Bsquare That's expected. It's the service hook Heroku registered on Github. See https://devcenter.heroku.com/articles/github-integration#enabling-github-integration — Schwern, Dec 06 '18 at 16:52
@Schwern Hi, it's been a long time, but were you able to find a solution ? — Shrolox, Sep 14 '21 at 09:25

score 0 · Answer 1 · answered Dec 05 '18 at 18:42

0

If you have the caches enabled as described here it does not seem like there is much more you can do to cache the installs. If this is a serious problem you could always point to local packages that you cloned from the remote repos to speed things up.

answered Dec 05 '18 at 18:42

Yehuda Schwartz

3,378
3
29
38

I don't think I have a way to enable caches, my understanding is they should just work. I don't call `bin/compile`. I just supply the buildpack URL and my understanding is Heroku is supposed to do the rest. Is there a problem with the buildpack? – Schwern Dec 05 '18 at 19:36
the documentation i linked to says `If the build pack does intend to use a cache, it should create the CACHE_DIR directory if it doesn’t exist.` i assume if caching was intended it would have cached it – Yehuda Schwartz Dec 05 '18 at 20:16

score -1 · Answer 2 · answered Dec 05 '18 at 09:32

-1

After various search and try, I bring good news.

The key issue is your errors in log files, like:

bash: /tmp/buildpacks/b5ae60a477f2ba582d00a4ba23348059d40dea69cea6d982947f7c6afa72d2b8d69252449d43cbcf97a23341fd57300a3fef3224ace95482e1c0deaf8a844e4d/export: No such file or directory

This auto-generated directory is obviously a cache directory one, which is generated automatically when not fixed.

Fore instance, this can be confirmed inside the heroku-buildpack-selenium buildpack, defined in your app.json file:

# parse and derive params
BUILD_DIR=$1
CACHE_DIR=$2
LP_DIR=`cd $(dirname $0); cd ..; pwd`

It is confirmed by this official documentation I found:

The contents of CACHE_DIR will be persisted between builds. You can cache the results of long processes like dependency resolution here to speed up future builds.

So the key solution is to define this CACHE_DIR variable within your installation; so I guess it is either directly in your app.json file, or directly in your GitHub Hook.

Let me know if you need further help.

answered Dec 05 '18 at 09:32

Bsquare ℬℬ

4,423
11
24
44

It looks like `CACHE_DIR` isn't an environment variable but a variable inside `bin/compile` that's populated from its 2nd argument. I have no control over that. All I do is supply the buildpack URL. I guess I could try defining a CACHE_DIR environment variable and see what happens. I've also [purged the cache](https://github.com/heroku/heroku-repo#purge-cache) just to see if that helps. I'll let you know next deploy, and I'll reup the bounty. – Schwern Dec 05 '18 at 19:43
Yes it is the second argument used when launching bin/compile but according to the documentation I think it is Heroku which specify this argument with an environment variable having the same name. – Bsquare ℬℬ Dec 05 '18 at 21:38
Could you copy/paste No example files which are under .git/hook in a clone of your repository? – Bsquare ℬℬ Dec 06 '18 at 07:31
There are no Git hooks in this project. The only way Git and Github are involved is Heroku is notified there's a new commit on master to deploy. – Schwern Dec 06 '18 at 18:32
Did you solve your issue? I'm still 100% sure it can be done either in your app.json file, or in your environment. Anyway, did you think about: On Stackoverflow you could give [up-vote](https://stackoverflow.com/help/privileges/vote-up) to people's helpful answers to thank them and select any one of the answer as [correct answer](https://stackoverflow.com/help/someone-answers) too out of all. – Bsquare ℬℬ Dec 10 '18 at 10:33

Can Selenium and Chrome buildpack installs be cached on HerokuCI?

2 Answers2