Reduce Duplications in Github Actions

Question

I have read and tested different things but didn't found a solution. Maybe I have to live with the boilerplate, but I am not yet done trying it.

I have the follow github actions workflow:

---
name: Code Quality Checks

on:
  push:
    branches: [main, development]
  pull_request:
    branches: [main, development]

jobs: 
  test:
    runs-on: ubuntu-latest
    needs: setup
    steps:
      - name: get repo
        uses: actions/checkout@v2
      - name: Set up Python 3.7
        uses: actions/setup-python@v1
        with:
          python-version: 3.7
      - name: Install poetry
        uses: snok/install-poetry@v1.0.0
        with:
          virtualenvs-create: true
          virtualenvs-in-project: true
      - name: Load cached venv
        id: cached-poetry-dependencies
        uses: actions/cache@v2
        with:
          path: .venv
          key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
      - name: Install dependencies
        run: poetry install
        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
      - name: Unit & Coverage test with pytest
        run: poetry run pytest

  check:
    runs-on: ubuntu-latest
    needs: setup
    steps:
      - name: get repo
        uses: actions/checkout@v2
      - name: Set up Python 3.7
        uses: actions/setup-python@v1
        with:
          python-version: 3.7
      - name: Install poetry
        uses: snok/install-poetry@v1.0.0
        with:
          virtualenvs-create: true
          virtualenvs-in-project: true
      - name: Load cached venv
        id: cached-poetry-dependencies
        uses: actions/cache@v2
        with:
          path: .venv
          key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
      - name: Install dependencies
        run: poetry install
        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
      - name: Check style with flake8
        run: poetry run flake8 boxsup_pytorch/ tests/

As you might see there is a lot of boilerplate in both jobs. I tested 'composition' which has issues with the cached variables. I also tinkert around with upload and download artifacts, but since the setup-python action uses python binary outside my cwd I was thinking uploading the complete runner is not a good idea.

Any working solutions for my scenario?

My previous Ideas was mostly based on In a github actions workflow, is there a way to have multiple jobs reuse the same setup?

Answer 1

What you are looking for is the concept of reusable workflow on Github Actions.

Reusing workflows avoids duplication. This makes workflows easier to maintain and allows you to create new workflows more quickly by building on the work of others, just as you do with actions. Workflow reuse also promotes best practice by helping you to use workflows that are well designed, have already been tested, and have been proved to be effective.

Just be aware of the limitations:

• Reusable workflows can't call other reusable workflows.
• Reusable workflows stored within a private repository can only be used by workflows within the same repository.
• Any environment variables set in an env context defined at the workflow level in the caller workflow are not propagated to the called workflow.
• The strategy property is not supported in any job that calls a reusable workflow.

Here is the documentation step by step to create a reusable workflow.

EDIT: And if you wonder what's the differences between composite actions and reusable workflows, I recommend reading this Github blog post

Answer 2

You could use a composite action that takes the cache key as an input; something like this (with updated action versions as of today):

name: Install Python and Poetry

inputs:
  python-version:
    description: Python version
    required: false
    default: 3.7
  cache-key:
    description: Key to use for cache action
    required: true

runs:
  using: composite
  steps:
    - name: Set up Python
      uses: actions/setup-python@v3.1.2
      with:
        python-version: ${{ inputs.python-version }}
    - name: Install poetry
      uses: snok/install-poetry@v1.3.1
      with:
        virtualenvs-in-project: true
    - name: Load cached venv
      id: cache
      uses: actions/cache@v3.0.2
      with:
        path: .venv
        key: ${{ inputs.cache-key }}
    - name: Install dependencies
      if: '! steps.cache.outputs.cache-hit'
      shell: bash
      run: poetry install

which replaces the steps 2-5 in your jobs. It also lets you specify the Python version, but defaults to 3.7 when that is omitted.

The workflow would then look something like this:

name: Code Quality Checks

on:
  push:
    branches:
      - main
      - development
  pull_request:
    branches:
      - main
      - development

jobs:
  test:
    runs-on: ubuntu-20.04
    needs: setup
    steps:
      - name: Get repo
        uses: actions/checkout@v3.0.2
      - name: Install Python and Poetry
        uses: ./.github/actions/poetry
        with:
          cache-key: env-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
      - name: Unit & coverage test with pytest
        run: poetry run pytest

  check:
    runs-on: ubuntu-20.04
    needs: setup
    steps:
      - name: Get repo
        uses: actions/checkout@v3.0.2
      - name: Install Python and Poetry
        uses: ./.github/actions/poetry
        with:
          cache-key: env-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
      - name: Check style with flake8
        run: poetry run flake8 boxsup_pytorch/ tests/

This assumes that the action.yml above lives in .github/actions/poetry, i.e., the .github directory would look like this:

.github
├── actions
│   └── poetry
│       └── action.yml
└── workflows
    └── checks.yml