Triggering Azure DevOps builds based on changes to sub folders

Question

I have a Visual Studio solution with multiple projects, each project is an individual microservice. It is very convenient to the development team to have all the services in the same solution, and git repo, as services can call each other.

Master.sln - SubFolderA - MicroserviceA.sln
           - SubFolderB - MicroserviceB.sln
           - SubFolderC - MicroserviceC.sln

I would, however, like to independently build/release the individual microservices in Azure DevOps when they change, so if ServiceA is the only service to change, then ServiceA is the only service built and deployed.

To this end I created a new build pipeline definition with "Path filters" set to trigger the build when the contents of a microservice folder change (so one path filter added per microservice to monitor).

My problem here is that when a build is triggered (based on a change to SubFolderA e.g.) I have no way to tell the build definition to only build the .sln file in SubFolderA.

I could create a separate build definition for each microservice and trigger each build on separate subfolders, but this would come at significant overhead, i.e. I would need to maintain 15 separate build definitions (same again for each branch I build), and the storage required on our self host build agent would now be NumberOfService x NumberOfBranchesBeingBuild x SizeOfRepo.

Is there a way to use a single Build Definition with git "Path filters" and multiple paths defined, which in turn kicks off multiple build instances and feeds the value of the path that triggered the build into the build definition and so telling the build instance which .sln file to build?

I hope that makes sense!

Answer 1

You can do like below

1. Create variables based on your microservices with the values "False"

E.g,MicroserviceAUpdated= "False",MicroserviceBUpdated= "False" etc.,

2. Add a Powershell script task at the begin of your build definition. The powershell script will do the following:

Get the changeset/commit in the build to check which files are changed.

• Update the MicroserviceAUpdated variable to "true" if only any files are changed under SubFolderA.
• Update the MicroserviceBUpdated variable to "true" if only any
  files are changed under SubFolderA.

So on....

3. Create separate build task for each microservice, configure the build tasks to run with custom conditions like below

For MicroserviceA build Task

"Custom conditions": and(succeeded(), eq(variables['MicroserviceAUpdated'], 'True'))

For MicroserviceB build Task

"Custom conditions": and(succeeded(), eq(variables['MicroserviceBUpdated'], 'True'))

So on...

This way MicoserviceTask will be skipped if the value of the variable is False

For Step 2

$files=$(git diff HEAD HEAD~ --name-only)
$temp=$files -split ' '
$count=$temp.Length
echo "Total changed $count files"
For ($i=0; $i -lt $temp.Length; $i++)
{
  $name=$temp[$i]
  echo "this is $name file"
  if ($name -like "SubFolderA/*")
    {
      Write-Host "##vso[task.setvariable variable=MicroserviceAUpdated]True"
    }
}

Answer 2

On the Triggers tab, there is an option to specify the path to the project you want to build. When that path is specified, only commits which contain modifications that match the include/exclude rules will trigger a build.

In my case this is a much better solution than the PowerShell script which still triggered builds and releases on all the projects spamming our Slack and filling with rubbish our project's history.

Answer 3

This post has helped me a lot so I wanted to add some useful modifications I've made to my process.

The first major problem I found is that this git diff command does not handle multiple commits at once.

git diff HEAD HEAD~ --name-only

HEAD~ only looks behind 1 commit, where as a single push could contain multiple commits at once.

I realized that I needed to do the diff between HEAD and the commit id since the pipeline last ran successfully.

git diff HEAD [commit id of last successful build] --name-only

This commit id is available by calling the Azure DevOps API at the /build/latest endpoint, sourceVersion.

$response = (Invoke-RestMethod -Uri $url -Method GET -Headers $AzureDevOpsAuthenicationHeader)
$editedFiles = (git diff HEAD $response.sourceVersion --name-only)

I also made a modification to the logic finding changed project / module folders. I did not want to have to modify my PowerShell script every time I added a new project by hardcoding my project names.

$editedFiles | ForEach-Object { 
    $sepIndex = $_.IndexOf('/')
    if($sepIndex -gt 0) {
        $projectName = $_.substring(0, $sepIndex)
        AppendQueueVariable $projectName
    }
}

AppendQueueVariable will maintain a list of all the changed projects to return to the pipeline.

Finally, I take the list of queued projects and pass them into my Maven multi-module build pipeline task.

mvn -amd -pl [list returned from PS task] clean install

Answer 4

Jayendran answer is very excellent! Here's a more PowerShell-y way to do step 2:

$editedFiles = git diff HEAD HEAD~ --name-only
$editedFiles | ForEach-Object {
    Switch -Wildcard ($_ ) {
        'SubFolderA/*' { Write-Output "##vso[task.setvariable variable=MicroserviceA]True" }
        # The rest of your path filters
    }
}

Answer 5

To complement deleb's answer, here is the YAML code to set up the path trigger:

trigger:
  branches:
    include:
    - master
  paths:
    include:
    - path/to/src*

Note that you need to have the branch trigger also to use the path trigger.

https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/azure-repos-git?view=azure-devops&tabs=yaml#paths

Edit: fixed path per Dariusz's comment

Answer 6

The solution is to have one azure-pipelines.yml per service which is in a sub-folder. Each of the azure-pipelines.yml inside the sub-folders must have the following trigger definition.

trigger:
  branches:
    include:
      - master
  paths:
    include:
      - <service subfolder name>/*

The paths -> include section of the yaml, tells azure pipelines to trigger only if there are changes in that particular path.

Its not necessary to give a different name to azure-pipelines.yml inside the sub-folders and the same name can be retained. Similarly, it is not necessary to add a azure-pipelines.yml to the root of the repo unless there is some code which needs to be built in the root outside the subfolders. In such cases, the following trigger section has to be added to the azure-pipelines.yml at the root of the repo.

trigger:
  branches:
    include:
      - master
  paths:
    exclude:
      - <service subfolder name 1>/*
      - <service subfolder name 2>/*

The paths -> exclude section excludes the subfolders which already contain their own azure-pipelines.yml file and gets triggered only when there is a change in the root of the repo outside the subfolders.

This blog explains monorepo pipelines in more detail.

Answer 7

In bash, you can do something like the following:

  - task: Bash@3
    displayName: 'Determine which apps were updated'
    inputs:
      targetType: 'inline'
      script: |
        DIFFS="$(git diff HEAD HEAD~ --name-only)"
        [[ "${DIFFS[@]}" =~ "packages/shared" ]] && echo "##vso[task.setvariable variable=SHARED_UPDATED]True"
        [[ "${DIFFS[@]}" =~ "packages/mobile" ]] && echo "##vso[task.setvariable variable=MOBILE_UPDATED]True"
        [[ "${DIFFS[@]}" =~ "packages/web" ]] && echo "##vso[task.setvariable variable=WEB_UPDATED]True"

Answer 8

Improvise on solution from Taul: if one project needs to be rebuilt when two or more subfolders are updated, we can use regex to detect all relevant changes:

Given MicroserviceA depends on folders SubfolderA, SubfolderB and BaseSubfolder

$editedFiles = git diff HEAD HEAD~ --name-only
$editedFiles | ForEach-Object {
    Switch -Regex ($_ ) {
        '(SubfolderA|SubfolderB|BaseSubfolder)(\/|$)' { Write-Output "##vso[task.setvariable variable=MicroserviceA]True" }
        # The rest of your path filters
    }
}