1

I see that Microsoft is likely going to move in the direction of shying away from Azure DevOps and more heavily leaning on GitHub Actions as a primary automation platform (speculation, not sure if it's true), so I am trying to move all of my automation off of DevOps onto GitHub Actions and when doing so I noticed that there are some lacking similarities.

In this specific case, I am wondering if there is an equivalent to Azure DevOps "Publish Pipeline Artifacts" task in GitHub Actions?

The closest thing I can find in GitHub Actions is "actions/upload-artifact@v2", however this more similarly resembles Azure DevOps' "Publish build artifacts". I get the use case and understand what I could use it for, but I want to see if I can upload an entire Pipeline/workflow in a package, rather than file by file.

In Azure DevOps, my pipeline runs in < 5-7 minutes because I can use the "Publish Pipeline Artifacts" task, but in GitHub Actions, I only have the "actions/upload-artifact@v2" action and now it takes up to 3 hours to do the same automation tasks. (Insane difference!). I think the added time is due to the upload/publish task in GitHub Actions going file by file whereas in Azure DevOps, the upload/publish task somehow condenses it all and it only takes ~1 minute for it to finish.

Any/All help is greatly appreciated! My Google Fu is not coming up with anything atm.

aseb
  • 274
  • 2
  • 11
  • 1
    I think if you zip your files before uploading it should greatly increase speed. – riQQ Aug 03 '21 at 18:30
  • Any recommendations on how to do so? So if i want to publish and then download it later (within the same workflow), which actions do you recommend i use. I would essentially need to zip it, publish the zip file -> then download the zip file and unzip it right? – aseb Aug 03 '21 at 19:59
  • Yes, use a shell script for (un)zipping and the download/upload-artifact actions. – riQQ Aug 03 '21 at 20:21

1 Answers1

1

It is slow because:

GZip is used internally to compress individual files before starting an upload.

So this is not only the case due to the fact that each file is sent individually but each file is also compressed individually. Your best workaround at the moment would be compress whole directory as riQQ already wrote.

It can be done like this:

  - name: 'Tar files'
    run: tar -cvf my_files.tar /path/to/my/directory

  - name: 'Upload Artifact'
    uses: actions/upload-artifact@v2
    with:
      name: my-artifact
      path: my_files.tar

A big drawback is that now you need to each time unpack your artifact when you download it.

For more details please check this topic - Upload artifact dir is very slow

Krzysztof Madej
  • 32,704
  • 10
  • 78
  • 107
  • Thank you, I thought that zipping was pretty much my only option. Interesting that GitHub Actions doesn't have a prepackaged action that does this task similarly as Azure DevOps. – aseb Aug 04 '21 at 04:12
  • Here is for instance [zip action](https://github.com/marketplace/actions/easy-zip-files) but this is not builtin action. – Krzysztof Madej Aug 04 '21 at 04:21
  • I was able to do it with just the linux zip command. On a different note, do you have any suggestions as to how I would pass a published artifact to a completely separate workflow? Is there a way to do to that? Say I did compress and zip a directory then published it, how then would i be able to pick it up in a separate workflow? – aseb Aug 04 '21 at 04:57
  • You can use this action [`dawidd6/action-download-artifact@v2`](https://github.com/dawidd6/action-download-artifact) to download artifact from another workflow. – Krzysztof Madej Aug 04 '21 at 06:48