Azure Devops - run build only in the folder that have changes

Question

I have a some spark jobs which are written on scala and build using maven. Right now the follow a path like this

/src/job1/<<build data>>
/src/job2/<<build data>>
.....

This build data holds the code, pom.xml, tests and other thinks needed for each job pass to maven.

We have a root folder that has a father pom.xml that builds all the code, but I don't want to build all the code every time. I want to be able to test and build only the code for the jobs that changes.

The only way that I think is possible is to run build an tests only in the folders that changes was detected but I can't seem to do that without create one azure devops build for each folder, any ideas?

TL:DR - How to pass the path of folder that have changes in azure devops to the build process?

score 2 · Answer 1 · answered Feb 28 '20 at 11:58

How to pass the path of folder that have changes in azure devops to the build process?

We could add a powershell scripts to check the folder or path, like:

$editedFiles = git diff HEAD HEAD~ --name-only
echo "$($editedFiles.Length) files modified:"
$editedFiles | ForEach-Object {
   echo $_
    Switch -Wildcard ($_ ) {
        '/src/job1' { 
              Write-Output "##vso[task.setvariable variable=job1]True"
         }        
        '/src/job2' { 
              Write-Output "##vso[task.setvariable variable=job2]True" }
    }
}

This script sets variables which are then referenced in custom conditions in the pipeline build task or in the build pipeline:

and(succeeded(), eq(variables['job1'], 'True'))

You could check this similar thread for some details.

Hope this helps.

I saw that post, my problem is that I need this structure to work across a huge number of projects. Some of then has about 30-40 jobs running and I don't want to configure one by one. I believe you didn't answer my question but made possible to do it with git. Thanks — Luiz Fernando Lobo, Feb 28 '20 at 18:58
@LuizFernandoLobo, Yes, if you have a huge number of projects, this solution is really not very well. But if you do not want to configure those projects or jobs one by one, I am afraid there is no such way to resolve this issue at this moment. — Leo Liu, Feb 29 '20 at 04:38

score 1 · Answer 2 · answered Feb 27 '20 at 13:31

You can work around this by creating a script that will do the following:

Get the changeset/commit in the build to check which files are changed.
Based on the files changes, you define the logic to decide which folder it should build, run tests, etc.

This solution will require you to build all the logic to decide which folders to build and test.

score 0 · Accepted Answer · answered Mar 04 '20 at 19:12

Thanks for your inputs. I was able to do it with your inputs on git, I realized that we are using gitflow workflow and every time that we have pull request to the master branch (meaning deployment) a commit is created with starting "Merged PR", this way I can compare the latest two commits and check what changed.

How I'm doing:

    # This part uses this to get the latest two pull requests hashes
    $last_pr_commit=git log  --grep="^Merged PR" -n 1  --pretty=format:"%h" --abbrev=200
    $second_last_pr_commit=git log -n 1  --grep="^Merged PR"  --oneline --abbrev=200 $last_pr_commit^...HEAD~2 --pretty=format:"%h"

    # This get  the path of the files
    $result=git diff $last_pr_commit $second_last_pr_commit --name-only

And then I just build the ones that I'm interested.

    ForEach ($line in $($result -split "`r`n"))
    {
        # In my case I only need the two first parts of path
        $folderPath=$line.split('/')[0]+'/'+$line.split('/')[1]
        if ($line.contains('.scala')){
            # Running the tests and packaging
            mvn package $folderPath
        }
    }

This is powershell but it can work with bash changing the for loop.

Thanks

Azure Devops - run build only in the folder that have changes

3 Answers3