11

I have folder structure this way in source code. f1 f2 f3 f4

I have added gitcopy diff task in my pipeline which lists and copies files which are modified to a destination folder. Now, I want to have a conditional loop as powershell script to only zip those folders which have modified files with a particular names for example if files from f1 are modified..I want particular steps to be performed and so on.. How can I do it as a loop? edit: I have written my pipeline in this way. But its failing in publish step with errors as listed.

yaml: trigger:

none

pool:
  vmImage: 'windows-latest'

variables:
  FR1PHPPRDAPP1VFlag: false
  FR1PHPPRDAPP4VFlag: false
  FR1PHPPRDAPP5VFlag: false
  FR1PHPPRDSRE1VFlag: false
  FR1PHPPRDAPP7VFlag: false
  stages:
  -stage: Zipping modified folders
steps:

- powershell: |

      ## get the changed files
      $files=$(git diff HEAD HEAD~ --name-only)
      $temp=$files -split ' '
      $count=$temp.Length
      echo "Total changed $count files"
      For ($i=0; $i -lt $temp.Length; $i++)
        {
          
          $name=$temp[$i]
          echo "this is $name file"
          if ($name -like 'FR1PHPPRDAPP1V/*') 
          {
            cd $(Build.ArtifactStagingDirectory)
            mkdir Output -force
           
          Compress-Archive -Path $(system.defaultworkingdirectory)/FR1PHPPRDAPP1V -DestinationPath $(Build.ArtifactStagingDirectory)/Output/APP1V.zip
          
          ##set the flag variable FR1PHPPRDAPP1VFlag to true
          Write-Host "##vso[task.setvariable variable=FR1PHPPRDAPP1VFlag]true"
          }
          if ($name -like 'FR1PHPPRDAPP4V/*')
          {
            cd $(Build.ArtifactStagingDirectory)
            mkdir Output -force
            ##achive folder FR1PHPPRDAPP4V if it is changed.
          Compress-Archive -Path $(system.defaultworkingdirectory)/FR1PHPPRDAPP4V -DestinationPath $(Build.ArtifactStagingDirectory)/Output/APP4V.zip
          ##set the flag variable FR1PHPPRDAPP4VFlag to true
          Write-Host "##vso[task.setvariable variable=FR1PHPPRDAPP4VFlag]True"
          }
           if ($name -like 'FR1PHPPRDAPP5V/*')
          {
            cd $(Build.ArtifactStagingDirectory)
            mkdir Output -force
            ##achive folder FR1PHPPRDAPP5V if it is changed.
          Compress-Archive -Path $(system.defaultworkingdirectory)/FR1PHPPRDAPP5V -DestinationPath $(Build.ArtifactStagingDirectory)/Output/APP5V.zip
          ##set the flag variable FR1PHPPRDAPP5VFlag to true
          Write-Host "##vso[task.setvariable variable=FR1PHPPRDAPP5VFlag]True"
          }
            if ($name -like 'FR1PHPPRDSRE1V/*')
          {
            cd $(Build.ArtifactStagingDirectory)
            mkdir Output -force
            ##achive folder FR1PHPPRDSRE1V if it is changed.
          Compress-Archive -Path $(system.defaultworkingdirectory)/FR1PHPPRDSRE1V -DestinationPath $(Build.ArtifactStagingDirectory)/Output/SRE1V.zip
          ##set the flag variable FR1PHPPRDSRE1VFlag to true
          Write-Host "##vso[task.setvariable variable=FR1PHPPRDSRE1VFlag]True"
          }
            if ($name -like 'FR1PHPPRDAPP7V/*')
          {
            cd $(Build.ArtifactStagingDirectory)
            mkdir Output -force
            ##achive folder FR1PHPPRDAPP7V if it is changed.
          Compress-Archive -Path $(system.defaultworkingdirectory)/FR1PHPPRDAPP7V -DestinationPath $(Build.ArtifactStagingDirectory)/Output/APP7V.zip
          ##set the flag variable FR1PHPPRDAPP7VFlag to true
          Write-Host "##vso[task.setvariable variable=FR1PHPPRDAPP7VFlag]True"
          }
        }
- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)/Output'
    ArtifactName: 'scripts-f2p'
    publishLocation: 'Container'
  condition: and(succeeded(), or(eq(variables.FR1PHPPRDAPP1VFlag, true),eq(variables.FR1PHPPRDAPP4VFlag, true),eq(variables.FR1PHPPRDAPP5VFlag, true),eq(variables.FR1PHPPRDSRE1VFlag, true),eq(variables.FR1PHPPRDAPP7VFlag, true)))
priya
  • 391
  • 1
  • 5
  • 24

4 Answers4

20

You can directly run below git commands in the powershell task to check the changed files. It is much easier than Rest api.

git diff-tree --no-commit-id --name-only -r $(Build.SourceVersion)

When you get the changed files, you can use the zip the changed folders directly in the powershell task using Compress-Archive command: See below example:

Compress-Archive -Path C:\f1 -DestinationPath f1.zip

If you want some particular steps to be performed based on the the changed folders. You can define the flag variables and using the logging commands in the powershell scripts to set the flags to true. And then use the condtions for the following steps.

See below full example scripts:

##set flag variables to indicate if the folder is changed.

variables:
  f1Flag: false
  f2Flag: false
  f3Flag: false
  
steps:

- powershell: |
      ## get the changed files
      $files=$(git diff-tree --no-commit-id --name-only -r $(Build.SourceVersion))
      $temp=$files -split ' '
      $count=$temp.Length
      echo "Total changed $count files"
     
      For ($i=0; $i -lt $temp.Length; $i++)
      {
        $name=$temp[$i]
        echo "this is $name file"
        if ($name -like 'f1/*')  #if f1 is a subfolder under a folder use "- like '*/f1/*'"
        { 
          ##achive folder f1 if it is changed.
          ##Compress-Archive -Path $(system.defaultworkingdirectory)/f1 -DestinationPath $(Build.ArtifactStagingDirectory)/f1.zip
          
          ##set the flag variable f1Flag to true
          Write-Host "##vso[task.setvariable variable=f2Flag]true"
        }
        if ($name -like 'f2/*')
        {
          ##achive folder f2 if it is changed.
          ##Compress-Archive -Path $(system.defaultworkingdirectory)/f2 -DestinationPath $(Build.ArtifactStagingDirectory)/f2.zip
          ##set the flag variable f2Flag to true
          Write-Host "##vso[task.setvariable variable=f2Flag]True"
        }
      }
      ## create a temp folder to hold the changed files
      New-Item -ItemType directory -Path $(system.defaultworkingdirectory)\temp

      foreach($file in $temp){
        if(Test-Path -path $file){
        Copy-Item -Path $file -Destination $(system.defaultworkingdirectory)\temp
        }
      }
      ## zip the temp folder which only have the changed files
      Compress-Archive -Path $(system.defaultworkingdirectory)\temp\* -DestinationPath $(Build.ArtifactStagingDirectory)\changedfiles.zip

Then you can use the condition for some particular steps just as Krzysztof mentioned

condition: and(succeeded(), or(eq(variables.f1Flag, true),eq(variables.f2Flag, true),eq(variables.f3Flag, true)))

See the answer to this thread for more information.

Update:

steps:

- powershell: |
     #get the changed files
     ....

        
- task: PublishBuildArtifacts@1
  inputs:
     PathtoPublish: '$(Build.ArtifactStagingDirectory)/Output'
     ArtifactName: 'drop'
     publishLocation: 'Container'
  condtion: and(succeeded(), or(eq(variables.f1Flag, true),eq(variables.f2Flag, true),eq(variables.f3Flag, true)))
Levi Lu-MSFT
  • 27,483
  • 2
  • 31
  • 43
  • Hi, could u check my edit? I have made changes to the pipeline.Now its failing in the condition statement – priya Dec 02 '20 at 07:07
  • @priya You can must use `stage` under `stages` section. you can use condtions for steps. no need to use stage. See above update – Levi Lu-MSFT Dec 02 '20 at 07:18
  • @priya How was it going? Did it work out? – Levi Lu-MSFT Dec 02 '20 at 09:20
  • @ lEVI, thankyou..could u pls tell me how can I add multiple conditions here? I want the publish task to run when either f1 or f2 or f3 flag turns true.. – priya Dec 02 '20 at 09:49
  • 1
    @priya You can use `or` expression. See above updated. Check [here](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops#or) for more information. – Levi Lu-MSFT Dec 02 '20 at 09:56
  • @ievi, thankyou.. but when am using this pipeline along with the changes..all the folder contents are coming..I just want only the modified file to be coming in the artifact download...pls check my whole pipeline in the edit – priya Dec 02 '20 at 11:05
  • @priya. You can just copy the modified files to a folder and then zip it. See above updated scripts in powershell task. – Levi Lu-MSFT Dec 03 '20 at 02:01
  • @Levi..thankyou very much for the help..just realized this solution works well if I have modified/added files..but build is failing when I renamed the file name..could you suggest me what changes can we make to even zip the files which got renamed? – priya Dec 03 '20 at 06:30
  • @priya. you can use `git diff-tree --no-commit-id` commands. And check if the file exists using `Test-Path -path`. See above updated scripts. – Levi Lu-MSFT Dec 03 '20 at 06:41
  • @Levi..this helped very much..Thankyou Levi :) – priya Dec 03 '20 at 07:02
  • 6
    it looks like this only works if you do a single commit per push, otherwise the build.sourceversion is only one of the commits that were pushed. Has anyone figured out a way to handle a multi-commit push? – Justin Nov 05 '21 at 21:07
  • it also appears that if the previous build fails, it only provides the commit id of the current attempt, not the previous attempts. i found a way to get a set of commits from a starting version, but i need to know the commit id of the previous succesful build. – Justin Nov 06 '21 at 03:42
  • 1
    I made some edits to handle using this in a pull request and with file names with spaces. In case the edits don't get added: Add a variable `isMain: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]`, `if ("${{ variables.isMain }}" -eq "true") { $changedFiles=$(git diff-tree --no-commit-id --name-only -r $(Build.SourceVersion)) } else { $changedFiles=$(git diff --name-only origin/main) }`, `Foreach ($name in $changedFiles)`. – Justin Harris Jan 13 '22 at 15:57
  • 1
    Oops it should be `if ("$(isMain)" -eq "true") ...`. – Justin Harris Jan 13 '22 at 19:46
  • @Justin You may be able to use the (Cache)[https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/cache?view=azure-devops] task to store the built commit id [as a last step when the build succeeds] and then do the git diff against that on subsequent runs. – Joshua Enfield Mar 18 '22 at 15:47
  • @LeviLu-MSFT is it possible to publish the changed files artifact to remote FTP server using above yml script? – Slimshadddyyy May 10 '22 at 20:02
  • 1
    If, like me, you're confused why this doesn't work in the pipeline but works locally when you test it, it might be because the default clone behaviour changed in the pipeline setup; it now defaults to shallow clone, with a depth of 1, so therefore you need to explicitly override this in your checkout step to get the commit before HEAD: `fetchDepth: 0` – Xyon Aug 04 '23 at 09:27
6

If you are creating a pipeline that checks files changed in an Azure Devops PR, you can create a Bash or Powershell task and use this command to get a list of the files changed:

Bash

git diff --name-only @~ @

Powershell

git diff --name-only HEAD~ HEAD

The reason this works is that for an Azure Repos Git PR, in the pipeline's local branch the git history contains one merge commit for all of the changes in the PR. It's just one merge commit regardless of how many commits were contained in the PR.

Max Nosik
  • 3
  • 1
TampaHaze
  • 1,075
  • 9
  • 6
1

I managed to get changed files using the following powershell script using azure devops REST API:

# remove 'refs/head' that is prefixed to branch names
$sourceBranch = "$(System.PullRequest.SourceBranch)"
$sourceBranch = $sourceBranch.Replace("refs/heads/", "")
$targetBranch = "$(System.PullRequest.TargetBranch)"
$targetBranch = $targetBranch.Replace("refs/heads/", "")

# auth headers
$headers=@{Authorization = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("azdo:$(System.AccessToken)")) }

# get PR diff changes from API
$response = (Invoke-WebRequest -Uri "https://{instance}[/{team-project}]/_apis/git/repositories/$(Build.Repository.ID)/diffs/commits?baseVersion=$targetBranch&targetVersion=$sourceBranch&api-version=5.1" -Method GET -Headers $headers  | ConvertFrom-Json)

# get path to changed files only, join them by spaces
$paths = $response.changes | Where-Object { -not $_.item.isFolder -or $_.item.isFolder -eq $false } | ForEach-Object { $_.item.path }
AzGhort
  • 31
  • 4
0

There is nothing out of the box solution. But you can use REST API Commits - Get Changes call to check that:

GET https://dev.azure.com/fabrikam/_apis/git/repositories/278d5cd2-584d-4b63-824a-2ba458937249/commits/be67f8871a4d2c75f13a51c1d3c30ac0d74d4ef4/changes?top=2&skip=10&api-version=5.0

{
  "changeCounts": {
    "Add": 456
  },
  "changes": [
    {
      "item": {
        "gitObjectType": "blob",
        "path": "/MyWebSite/MyWebSite/favicon.ico",
        "url": "https://dev.azure.com/fabrikam/_apis/git/repositories/278d5cd2-584d-4b63-824a-2ba458937249/items/MyWebSite/MyWebSite/favicon.ico?versionType=Commit"
      },
      "changeType": "add"
    },
    {
      "item": {
        "gitObjectType": "tree",
        "path": "/MyWebSite/MyWebSite/fonts",
        "isFolder": true,
        "url": "https://dev.azure.com/fabrikam/_apis/git/repositories/278d5cd2-584d-4b63-824a-2ba458937249/items/MyWebSite/MyWebSite/fonts?versionType=Commit"
      },
      "changeType": "add"
    }
  ]
}

If you use gitcopy diff task and you didn't flatten destination folder you should analyze you destination folder using command line - powershell/bashh - and based on this set variables.

$f1Changed = ((Get-ChildItem -Path .\destination\f1\ | Measure-Object).Count -gt 0)

Then you can analyze this response and check what was changed. Based on that you can set variables using logging command:

Write-Host "##vso[task.setvariable variable=f1changed;]$f1Changed"

And then use this variable in condition on your step here zip this particular folder.

 condition: and(succeeded(), eq(variables.f1changed, true))
Krzysztof Madej
  • 32,704
  • 10
  • 78
  • 107
  • I am using git copy diff task for this purpose..now I have the changed files.But I want to perform zip operation on changed files which can be part of any of the folders..could u revisit my question please..I have edted it – priya Dec 01 '20 at 10:45
  • Please check me edit. I added there powershell script to check if there are files in destination/f1 folder and then used this in logging command. – Krzysztof Madej Dec 01 '20 at 11:33
  • @krzystof Hi thanks for suggesting...could you check my edit please? I do not need the whole folder to be zipped..I jst need the changed files to get zipped in the destination folder..could u suggest any changes? – priya Dec 02 '20 at 12:50
  • Well in this case please just zip your destination folder when you copied your just modified files. I missed that part that you want to copy just modified files. – Krzysztof Madej Dec 02 '20 at 12:58
  • I am not using copy diff task here..using powershell task to get the modified files through git diff command. check here https://stackoverflow.com/q/65088433/13460189 – priya Dec 02 '20 at 13:04
  • In this case since you know what was changed just copy this files to folder which you later zipped. – Krzysztof Madej Dec 02 '20 at 13:06