Github actions: How to cache dependencies between workflow runs of different branches?

Question

I have a CI that runs at every pull request created and at every push of a new commit. This CI installs Python dependencies and then run some tests. I use two separate requirements.txt files because one of them contains heavier packages and they are handled differently in Docker. I am trying to use the actions/cache@v2 action to cache the dependencies but from what I could understand, it only caches between runs in the same branch. So when I create a new PR, for example, cache is not detected from another branch and everything is installed from scratch. Is there a way to cache dependencies across workflow runs? So the cache created by the CI in one branch can be used by another branch if nothing was changed in the requirements?

Looking at the logs of the workflow that ran in two different branches the cache key is the same:

Workflow in branchA

Cache not found for input keys: /opt/hostedtoolcache/Python/3.8.12/x64-03a86b868f006751e123da18168c989ab4c3c2713de4f5c87cf732ffbb6fb4ae-cd1b416332d9d5b55f413e2bd74c2efce6107aef1ce3f497fa5a81b9abc83deb

Workflow in branchB

Cache not found for input keys: /opt/hostedtoolcache/Python/3.8.12/x64-03a86b868f006751e123da18168c989ab4c3c2713de4f5c87cf732ffbb6fb4ae-cd1b416332d9d5b55f413e2bd74c2efce6107aef1ce3f497fa5a81b9abc83deb

This is my workflow:

name: ci

on:
  pull_request:
    types: [opened, synchronize]
    branches-ignore:
      - "master"
      - "staging"

  push:
    branches-ignore:
      - "master"
      - "staging"

jobs:
  run-tests:
    name: Run tests
    runs-on: ubuntu-latest

    steps:
      - name: Set up Python 3.8
        uses: actions/setup-python@v2
        with:
          python-version: 3.8

      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Cache dependencies
        uses: actions/cache@v2
        with:
          path: ${{ env.pythonLocation }}
          key: ${{ env.pythonLocation }}-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-ml.txt') }}

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          python -m pip install -r requirements.txt
          python -m pip install -r requirements-ml.txt

@LearningToNLP yes I did! So the thing is that because all of our branches are created against a "staging" branch, any cache created in the base branch will be available to the branches created off of it. So I created a new job in the staging CI that will update the cache in parallel everytime there's a new push event. But the good thing is that the cache is updated only if really needed. I'll post as answer my solution — everspader, Feb 10 '22 at 07:28

score 4 · Accepted Answer · edited Sep 16 '22 at 15:07

The solution to my problem was very silly but effective. I created a specific workflow for the staging branch for every commit push that will update the cache (if needed). This is helpful because all the development branches are created off of the staging branch, and every cache available in a base branch is also available to the derived branches. So at the end, the solution is like the original post but with the event's updated.

From github's docs:

A workflow can access and restore a cache created in the current branch, the base branch (including base branches of forked repositories), or the default branch (usually main). For example, a cache created on the default branch would be accessible from any pull request. Also, if the branch feature-b has the base branch feature-a, a workflow triggered on feature-b would have access to caches created in the default branch (main), feature-a, and feature-b.

name: update-cache

on:
  push:
    branches:
      - "staging"

jobs:
  # This workflow builds the Python package dependencies every time that the requirements
  # files are modified and store it in cache to be accessible by all the CI in all other
  # branches.
  build-cache:
    name: Build Cache
    runs-on: ubuntu-latest

    steps:
      - name: Set up Python 3.8
        uses: actions/setup-python@v2
        with:
          python-version: 3.8

      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Cache dependencies
        uses: actions/cache@v2
        id: cache
        with:
          path: ${{ env.pythonLocation }}
          key: ${{ env.pythonLocation }}-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-ml.txt') }}
          restore-keys:
            ${{ env.pythonLocation }}-

      - if: steps.cache.outputs.cache-hit != 'true'
        name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          python -m pip install -r requirements.txt
          python -m pip install -r requirements-ml.txt

Github actions: How to cache dependencies between workflow runs of different branches?

1 Answers1

Linked