Releasing only changed files with Pipelines

Question

I am working on setting up a Azure DevOps project to update our website.
I set up a test website, and got it all working so that it would publish automatically whenever I did a Git Push.

The problem I'm having is that the test website has 2 files, and the real website has many many more, totalling in a little over 500MB.

I'm hoping there is a way to get it to only push out the files that changed, and not every single file.

My build pipeline is using the following script:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:

- task: ArchiveFiles@2
  displayName: 'ArchiveFiles'
  inputs:
    rootFolderOrFile: '$(System.DefaultWorkingDirectory)'
    includeRootFolder: false

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifacts: drop'

And the release pipeline is doing a IIS Web App Deploy.

The preview feature "Pipeline artifacts" has a filtering mechanism (.artifactignore) - https://learn.microsoft.com/en-us/azure/devops/pipelines/artifacts/pipeline-artifacts?view=azure-devops&tabs=yaml Is it helpful? — Amittai Shapira, Jul 31 '19 at 19:10
@AmittaiShapira It looks like it has to be set up to ignore specific files, even if they change, and I want it to ignore all files, unless they change. If that makes sense. Like, if file1.html changed, then push it over, but file2.html stayed the same, so don't worry about it. — AndyD273, Jul 31 '19 at 19:20
Yes, makes sense... looking at the source code it looks like this task is using Robocopy internally. My experience with the CopyFiles task (which also use Robocopy) is that it ignores files that weren't changed... Source code for the task is in GitHub: https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/PublishBuildArtifactsV1 — Amittai Shapira, Jul 31 '19 at 19:52
@LeoLiu-MSFT I haven't yet figured out how to do that with powershell. I played around with the CopyFiles@2 task, but I'm running into problems, and I think I'm missing some understanding on how everything works. — AndyD273, Aug 06 '19 at 13:15

score 3 · Answer 1 · answered Jan 23 '21 at 14:26

We did face the same issue and had to check a few solutions on Stackoverflow and Google and finally worked out on the following solution, which helped us pick only the last modified files based on GIT and then publish only those files instead of pushing the whole lot.

    # HTML
    # Archive your static HTML project and save it with the build record.
    # Add steps that build, run tests, deploy, and more:
    # https://aka.ms/yaml
    # Updated to pick only modified files and push them for publish by adding "task: PowerShell" (to optimize CICD process)

    trigger:
    - master

    pool:
    vmImage: 'ubuntu-latest'

    steps:
    - task: PowerShell@2
    inputs:
    targetType: 'inline'
        #We fist check the latest modified files based on GIT DIFF command
        #Then we loop through the files using files-split
        #Next we split the file URL to get the file name and parent folder structure of the file separately
        #Based on the file path, we check the path in output directory and create it if its not present
        #Once the folder structure is available in the destination, we then copy the file to that location using copy-item command
        #Force parameter is used copy-item to overwrite any existing files

        script: $files=$(git diff HEAD HEAD~ --name-only);
        $temp=$files-split ' '; 
        $count=$temp.Length;
        echo "Total changed $count files";
        For ($i=0; $i -lt $temp.Length; $i++)
        {
            $name=$temp[$i];
            echo "Modified File -  $name file";
            $filepath = ($name.Split('/'))[0..($name.Split('/').count-2)] -join '/';
            echo "File path - $filepath";          
            $destFile = Join-Path $(Build.ArtifactStagingDirectory) $name;
            $destinationpath = Split-Path $destFile ;
            echo "Destination path - $destinationpath";
            if (!(Test-Path -Path $destinationpath)) {
                New-Item $destinationpath -ItemType Directory
            }
            Copy-Item $name -Destination $destFile -Recurse -Force 
     
        }

        Get-ChildItem -Path $(Build.ArtifactStagingDirectory) -Recurse -Force

    - task: ArchiveFiles@2
    inputs:
        rootFolderOrFile: '$(Build.ArtifactStagingDirectory)'
        includeRootFolder: false

    - task: PublishBuildArtifacts@1

Please note that the code above works perfectly fine, only only copy pasting due to tabs/spaces at the start of every line, we need to validate the code before actually running the build.

Happy coding..!

Hi! This is exactly what I am after. However, when I run this in the pipeline the step fails with error `fatal: ambiguous argument 'HEAD~': unknown revision or path not in the working tree.` Is this something you have seen before? — mr3k, Oct 20 '22 at 08:15

score 2 · Answer 2 · edited Aug 02 '19 at 12:49

2

Releasing only changed files with Pipelines

There is no such out of box method to only pick the modified files when using copy task or PublishBuildArtifacts.

As workaround, we could add powershell task to deletes all files (recursive) which have timestamp that is < (Now - x min), with this way Artifact directory contains of ONLY CHANGED files.

Check the details here: TFS 2017 - how build/deliver only changed files?.

I tried to develop this script using my lame PowerShell technology, but I did not success.

Hope this helps.

edited Aug 02 '19 at 12:49

Amittai Shapira

3,749
1
30
54

answered Aug 02 '19 at 08:12

Leo Liu

71,098
10
114
135

My experience with the Copy File Task is that it'll only replace files that changed in the target directory... However, as you mentioned, you can always use Robocopy (or similar) with powershell task – Amittai Shapira Aug 02 '19 at 12:48

score 2 · Answer 3 · answered Oct 26 '20 at 20:41

I was able to create an ADO powershell task based on the suggestion of another answer in this thread.

  - task: PowerShell@2
    inputs:
      targetType: 'inline'
      script: 'get-childitem $(build.artifactStagingDirectory) -recurse | Foreach { 
        $lastupdatetime=$_.LastWriteTime; 
        $nowtime=get-date; 
        if(($nowtime - $lastupdatetime).totalhours -le 24) { 
          Write-Host $_.fullname 
        } 
        else{ 
          remove-item $_.fullname -Recurse -Force 
        } 
      }'
      failOnStderr: true
      workingDirectory: '$(build.artifactStagingDirectory)'

The script portion of the task should be in one line, but I expanded it for easier reading.

score 0 · Answer 4 · edited Nov 08 '22 at 10:08

0

If you are getting error "unknown revision" then increase the shallow fetch depth of your Azure DevOps YAML pipeline. See the official documentation.

edited Nov 08 '22 at 10:08

Adriaan

17,741
7
42
75

answered Nov 08 '22 at 10:07

Sagar Jaiswal

23
4

Releasing only changed files with Pipelines

4 Answers4