3

Just a pretty stand curl command call an S3 end point for download using all default values. On a mac, or on a PC using command line I get 103MBsec if cached on cdn and 80mbsec otherwise. Same command, same bucket, same object, using "curl.exe" and I get 1MBSec when call through powershell. I guess powershell does something different that make it's totally slow? I tried using newest curl binary but still the same. I guess I am misunderstanding what powershell is doing when I use a curl command

curl.exe yourfileonS3 >> output.bin
user1456958
  • 59
  • 1
  • 8

2 Answers2

4

To complement briantist's helpful answer:

  • In PowerShell, the redirection operators > and >> are in effect aliases of Out-File and Out-File -Append.

  • > and >> are therefore not mere byte-stream conduits, and, in fact, PowerShell as of v7.2 does not support sending raw byte output to a file.

    • Instead, PowerShell invariably decodes output from any external program as text ([string] instances), based on the character encoding reported by [Console]::OutputEncoding] and then, on saving to the target file with Out-File (possibly via > / >>), re-encodes these strings, using that cmdlet's default character encoding (unless overridden with -Encoding in an explicit Out-File call).

    • Not only does this not preserve the external program's raw byte output, it adds significant overhead.

To get raw byte processing, call cmd.exe[1] and use its redirection operators:

cmd /c 'curl.exe yourfileonS3 >> output.bin'

See this answer for more information.

[1] On Unix-like platforms, use sh -c 'curl yourfileonS3 >> output.bin'

mklement0
  • 382,024
  • 64
  • 607
  • 775
  • 2
    Awesome! This was the missing piece I didn't have time to complete. And I suspect that not being able to handle raw bytes is a big deal when trying to download binary data into a file. Excellent answer as usual. – briantist Aug 15 '21 at 22:00
3

See mklement0's answer for full context on this (I recommend accepting that one!), and the important point that handling of byte streams in redirection is problematic and error prone in PowerShell and should be avoided.

So I looked into this and I believe the reason is that >> (file redirection) is the slow part.

I originally suspected you might be calling curl (which is aliased to Invoke-WebRequest in Windows PowerShell), but I was able to reproduce the speed difference between curl.exe directly in PowerShell vs cmd.exe, and measure it, this way:

# call curl.exe and do redirection in PowerShell
Measure-Command -Expression { curl.exe https://uploader.codecov.io/v0.1.0_6943/linux/codecov >> delme.bin }

del delme.bin

# call cmd.exe and do redirection there
Measure-Command -Expression { & cmd.exe /c 'curl.exe https://uploader.codecov.io/v0.1.0_6943/linux/codecov >> delme.bin' }

del delme.bin

This was enough to show a stark difference.

I also confirmed that this problem is a little bit worse in Windows PowerShell as opposed to later cross-platform versions (pwsh.exe). In Windows, with version 7.1.0, the same commands above still show a large difference.

briantist
  • 45,546
  • 6
  • 82
  • 127