I have workflow that needs to have a loop for the steps, which is perfect with strategy/matrix.
The only problem is that strategy/matrix needs to be set by a constant.
Is it possible to use strategy matrix with a output of a script?
name: tests
on: [push]
jobs:
test:
runs-on: ${{ ubuntu-latest }}
strategy:
fail-fast: false
matrix:
versions: $(./script.py)
steps:
- uses: actions/checkout@v2
.......
You can generate matrix in JSON in one job and set it to the second job.
GitHub added this feature in April: https://github.blog/changelog/2020-04-15-github-actions-new-workflow-features/
name: build
on: push
jobs:
job1:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- id: set-matrix
run: echo "::set-output name=matrix::{\"include\":[{\"project\":\"foo\",\"config\":\"Debug\"},{\"project\":\"bar\",\"config\":\"Release\"}]}"
job2:
needs: job1
runs-on: ubuntu-latest
strategy:
matrix: ${{fromJson(needs.job1.outputs.matrix)}}
steps:
- run: echo ${{ matrix.project }}
- run: echo ${{ matrix.config }}
First job sets output variable matrix to JSON that contains two configurations:
{
"include": [
{
"project": "foo",
"config": "Debug"
},
{
"project": "bar",
"config": "Release"
}
]
}
Equivalent in .yml:
job2:
strategy:
matrix:
include:
- project: foo
config: Debug
- project: bar
config: Release
Do not forget to escape quotes \" and print JSON in one line.
It detects changed files and runs build job for changed directories. If directory name starts with OS name, it uses that name as runs-on.
name: Build
on: [push, pull_request]
jobs:
generate-matrix:
name: Generate matrix for build
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v2
- name: Check changed files
id: diff
run: |
# See https://github.community/t/check-pushed-file-changes-with-git-diff-tree-in-github-actions/17220/10
if [ $GITHUB_BASE_REF ]; then
# Pull Request
git fetch origin $GITHUB_BASE_REF --depth=1
export DIFF=$( git diff --name-only origin/$GITHUB_BASE_REF $GITHUB_SHA )
echo "Diff between origin/$GITHUB_BASE_REF and $GITHUB_SHA"
else
# Push
git fetch origin ${{ github.event.before }} --depth=1
export DIFF=$( git diff --name-only ${{ github.event.before }} $GITHUB_SHA )
echo "Diff between ${{ github.event.before }} and $GITHUB_SHA"
fi
echo "$DIFF"
# Escape newlines (replace \n with %0A)
echo "diff=$( echo "$DIFF" | sed ':a;N;$!ba;s/\n/%0A/g' )" >> $GITHUB_OUTPUT
- name: Set matrix for build
id: set-matrix
run: |
# See https://stackoverflow.com/a/62953566/11948346
DIFF="${{ steps.diff.outputs.diff }}"
JSON="{\"include\":["
# Loop by lines
while read path; do
# Set $directory to substring before /
directory="$( echo $path | cut -d'/' -f1 -s )"
if [ -z "$directory" ]; then
continue # Exclude root directory
elif [ "$directory" == docs ]; then
continue # Exclude docs directory
elif [ "$path" == *.rst ]; then
continue # Exclude *.rst files
fi
# Set $os. "ubuntu-latest" by default. if directory starts with windows, then "windows-latest"
os="ubuntu-latest"
if [ "$directory" == windows* ]; then
os="windows-latest"
fi
# Add build to the matrix only if it is not already included
JSONline="{\"directory\": \"$directory\", \"os\": \"$os\"},"
if [[ "$JSON" != *"$JSONline"* ]]; then
JSON="$JSON$JSONline"
fi
done <<< "$DIFF"
# Remove last "," and add closing brackets
if [[ $JSON == *, ]]; then
JSON="${JSON%?}"
fi
JSON="$JSON]}"
echo $JSON
# Set output
echo "matrix=$( echo "$JSON" )" >> $GITHUB_OUTPUT
build:
name: Build "${{ matrix.directory }}" on ${{ matrix.os }}
needs: generate-matrix
strategy:
matrix: ${{fromJson(needs.generate-matrix.outputs.matrix)}}
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v2
- name: Build
run: |
cd ${{ matrix.directory }}
echo "${{ matrix.directory }} ${{ matrix.os }}"
Adding a new example, it was a really helpful answer. Thank you @ArtemSBulgakov !
This one uses Github strategy.matrix of Github Actions with fromJson to collect only the directories in a Pull Request with changes and make a Syntax Review and Format Review of Terraform using https://github.com/dflook/terraform-github-actions
---
name: Check Syntax
on: [pull_request]
jobs:
generate-matrix:
name: Generate matrix for build
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Check changed files
id: diff
run: |
# See https://github.community/t/check-pushed-file-changes-with-git-diff-tree-in-github-actions/17220/10
export DIFF=$( git diff --dirstat=files,0,cumulative ${{ github.event.pull_request.base.sha }} | awk -F ' ' '{print $2}' )
echo "$DIFF"
# Escape newlines (replace \n with %0A)
echo "::set-output name=diff::$( echo "$DIFF" | sed ':a;N;$!ba;s/\n/%0A/g' )"
- name: Set matrix for build
id: set-matrix
run: |
# See https://stackoverflow.com/a/62953566/11948346
DIFF="${{ steps.diff.outputs.diff }}"
JSON="{\"tfpaths\":["
# Loop by lines
while read path; do
# Add item to the matrix only if it is not already included
JSONline="\"$path\","
if [[ "$JSON" != *"$JSONline"* ]]; then
JSON="$JSON$JSONline"
fi
done <<< "$DIFF"
# Remove last "," and add closing brackets
if [[ $JSON == *, ]]; then
JSON="${JSON%?}"
fi
JSON="$JSON]}"
echo $JSON
# Set output
echo "::set-output name=matrix::$( echo "$JSON" )"
validate:
name: Check Terraform syntax on "${{ matrix.tfpaths }}"
needs: generate-matrix
strategy:
matrix: ${{fromJson(needs.generate-matrix.outputs.matrix)}}
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: terraform validate
uses: dflook/terraform-validate@v1
with:
path: ${{ matrix.tfpaths }}
check-format:
name: Check Terraform format on "${{ matrix.tfpaths }}"
needs: generate-matrix
strategy:
matrix: ${{fromJson(needs.generate-matrix.outputs.matrix)}}
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: terraform fmt
uses: dflook/terraform-fmt-check@v1
with:
path: ${{ matrix.tfpaths }}
I used @ArtemSBulgakov's solution as a starting point but got stuck trying to generate the matrix from some other output, rather than explicit json string.
If you are like me and want a way to feed the matrix from other outputs see below.
In this example, I want to fetch the latest pull requests from GitHub using the octokit/request-action action, then performing some checks on each pull request. Sounds pretty simple but transforming the output (i.e. steps.fetch.outputs.data) into something like this...
{
"includes": [{ "number": 1, "title": "my first pr " }]
}
...proved to be much harder than I expected. You may be more skilled at doing this with one of the available shell's GitHub provides but then you still have to somehow pass the output values to the run script and then back out again. If someone knows an easy way to do this, I would be happy to see it.
So I decided to create the nickofthyme/object-remap GitHub action to make this a little easier. I won't get into all the usages (see the README.md) but an example usage to set the matrix.includes using Object filters (i.e. .*.) would look like this...
name: 'PR Check'
on:
schedule:
- cron: '0 0 * * *' # once a day
jobs:
fetch:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.save.outputs.json }}
steps:
- name: Fetch GH pulls
id: fetch
uses: octokit/request-action@v2.x
with:
route: GET /repos/{repo}/pulls?state=open
repo: ${{ github.repository }}
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
- name: Store matrix
id: save
uses: nickofthyme/object-remap@v1
with:
include.*.number: ${{ toJSON(fromJSON(steps.fetch.outputs.data).*.number) }}
include.*.title: ${{ toJSON(fromJSON(steps.fetch.outputs.data).*.title) }}
include.*.username: ${{ toJSON(fromJSON(steps.fetch.outputs.data).*.user.login) }}
pr-checks:
name: "PR #${{ matrix.number }} - ${{ matrix.title }}"
runs-on: ubuntu-latest
needs: fetch
strategy:
matrix: ${{ fromJSON(needs.fetch.outputs.matrix) }}
fail-fast: false
steps:
- name: Echo pr number
run: echo "pr number: ${{ matrix.number }}"
- name: Echo title
run: echo "title: ${{ matrix.title }}"
- name: Echo username
run: echo "username: ${{ matrix.username }}"
For example, if this workflow were run on react's first 2 PRs
curl https://api.github.com/repos/facebook/react/pulls?per_page=2&direction=asc&state=all
The steps.fetch.outputs.data, omitting head, base and _links for brevity, would be...
[
{
"url": "https://api.github.com/repos/facebook/react/pulls/1",
"id": 6001916,
"node_id": "MDExOlB1bGxSZXF1ZXN0NjAwMTkxNg==",
"html_url": "https://github.com/facebook/react/pull/1",
"diff_url": "https://github.com/facebook/react/pull/1.diff",
"patch_url": "https://github.com/facebook/react/pull/1.patch",
"issue_url": "https://api.github.com/repos/facebook/react/issues/1",
"number": 1,
"state": "closed",
"locked": false,
"title": "Run each test in its own <iframe>",
"user": {
"login": "benjamn",
"id": 5750,
"node_id": "MDQ6VXNlcjU3NTA=",
"avatar_url": "https://avatars.githubusercontent.com/u/5750?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/benjamn",
"html_url": "https://github.com/benjamn",
"followers_url": "https://api.github.com/users/benjamn/followers",
"following_url": "https://api.github.com/users/benjamn/following{/other_user}",
"gists_url": "https://api.github.com/users/benjamn/gists{/gist_id}",
"starred_url": "https://api.github.com/users/benjamn/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/benjamn/subscriptions",
"organizations_url": "https://api.github.com/users/benjamn/orgs",
"repos_url": "https://api.github.com/users/benjamn/repos",
"events_url": "https://api.github.com/users/benjamn/events{/privacy}",
"received_events_url": "https://api.github.com/users/benjamn/received_events",
"type": "User",
"site_admin": false
},
"body": "This is not blocking the initial launch, so feel free to put it on the back-burner for now.\n\nThe Jasmine test harness still runs in the parent window and reports to PhantomJS via `window.callPhantom`, but each test `<iframe>` has its own copy of `react-test.js` and each individual test module is required in the global context of a separate `<iframe>`.\n\nThis gives us a significant approximation of the benefits of mocking, at least in terms of isolating tests from one another.\n\ncr @jeffmo @zpao\n",
"created_at": "2013-05-29T20:20:53Z",
"updated_at": "2014-07-16T22:39:07Z",
"closed_at": "2013-06-03T17:58:02Z",
"merged_at": "2013-06-03T17:58:02Z",
"merge_commit_sha": "7a72883d48e00854a41a1cdff99a2544c1721dcc",
"assignee": null,
"assignees": [],
"requested_reviewers": [],
"requested_teams": [],
"labels": [],
"milestone": null,
"draft": false,
"commits_url": "https://api.github.com/repos/facebook/react/pulls/1/commits",
"review_comments_url": "https://api.github.com/repos/facebook/react/pulls/1/comments",
"review_comment_url": "https://api.github.com/repos/facebook/react/pulls/comments{/number}",
"comments_url": "https://api.github.com/repos/facebook/react/issues/1/comments",
"statuses_url": "https://api.github.com/repos/facebook/react/statuses/603c9ef6a8d70d3cf29ee9d0a9d7969abce48ac4",
"head": {},
"base": {},
"_links": {},
"author_association": "CONTRIBUTOR",
"auto_merge": null,
"active_lock_reason": null
},
{
"url": "https://api.github.com/repos/facebook/react/pulls/2",
"id": 6002192,
"node_id": "MDExOlB1bGxSZXF1ZXN0NjAwMjE5Mg==",
"html_url": "https://github.com/facebook/react/pull/2",
"diff_url": "https://github.com/facebook/react/pull/2.diff",
"patch_url": "https://github.com/facebook/react/pull/2.patch",
"issue_url": "https://api.github.com/repos/facebook/react/issues/2",
"number": 2,
"state": "closed",
"locked": false,
"title": "[docs] Fix button links on bottom of home",
"user": {
"login": "paulshen",
"id": 2266187,
"node_id": "MDQ6VXNlcjIyNjYxODc=",
"avatar_url": "https://avatars.githubusercontent.com/u/2266187?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/paulshen",
"html_url": "https://github.com/paulshen",
"followers_url": "https://api.github.com/users/paulshen/followers",
"following_url": "https://api.github.com/users/paulshen/following{/other_user}",
"gists_url": "https://api.github.com/users/paulshen/gists{/gist_id}",
"starred_url": "https://api.github.com/users/paulshen/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/paulshen/subscriptions",
"organizations_url": "https://api.github.com/users/paulshen/orgs",
"repos_url": "https://api.github.com/users/paulshen/repos",
"events_url": "https://api.github.com/users/paulshen/events{/privacy}",
"received_events_url": "https://api.github.com/users/paulshen/received_events",
"type": "User",
"site_admin": false
},
"body": "The buttons on the index were pointing at wrong paths.\n",
"created_at": "2013-05-29T20:31:39Z",
"updated_at": "2014-06-27T04:39:06Z",
"closed_at": "2013-05-29T20:32:25Z",
"merged_at": "2013-05-29T20:32:25Z",
"merge_commit_sha": "9aa4d2bc27c38b01c9c8f3436bd729d5e656cb1b",
"assignee": null,
"assignees": [],
"requested_reviewers": [],
"requested_teams": [],
"labels": [],
"milestone": null,
"draft": false,
"commits_url": "https://api.github.com/repos/facebook/react/pulls/2/commits",
"review_comments_url": "https://api.github.com/repos/facebook/react/pulls/2/comments",
"review_comment_url": "https://api.github.com/repos/facebook/react/pulls/comments{/number}",
"comments_url": "https://api.github.com/repos/facebook/react/issues/2/comments",
"statuses_url": "https://api.github.com/repos/facebook/react/statuses/c5b4fe9e88a9a3b43cfd9b7e5383096bd9e213ef",
"head": {},
"base": {},
"_links": {},
"author_association": "CONTRIBUTOR",
"auto_merge": null,
"active_lock_reason": null
}
]
And the value of steps.save.outputs.json (aka needs.fetch.outputs.matrix) would be...
{
"includes": [
{
"state": "closed",
"title": "Run each test in its own <iframe>",
"username": "benjamn"
},
{
"number": 2,
"title": "[docs] Fix button links on bottom of home",
"username": "paulshen"
}
]
}
...which can easily be passed to jobs.<job_id>.strategy.matrix which would trigger two pr-checks jobs.
One final note: I attempted to just pass the matrix array of values to
jobs.<job_id>.strategy.matrix.includesbut this fails becausematrix.includesdoes not accept GitHub expressions as a value. So nesting values inincludesis the way to go!
