6

I am currently writing a script in Powershell that allows copying a folder in a SVN repository to another while preserving the history. An example of such command is:

svnadmin.exe dump $FromRepoPath `
    | svndumpfilter.exe include --drop-empty-revs --renumber-revs --preserve-revprops $Folder `
    | svnadmin.exe load --ignore-uuid $ToRepoPath

This causes an very high memory usage in Powershell. It appears that Powershell first executes svnadmin.exe and buffers the stdout from SVN admin, then executes svndumpfilter and buffers that output and finally executes svnadmin.exe.

I can work around it by creating a seperate batch file:

@echo off
svnadmin.exe dump %1 | svndumpfilter.exe include --drop-empty-revs --renumber-revs --preserve-revprops %2 | svnadmin.exe load --ignore-uuid %3

And then calling it from Powershell:

cmd.exe /c "SvnHelper.cmd $FromRepoPath $Folder $ToRepoPath"

But this feels like a nasty and unnecessary workaround.

Is there any way to tell Powershell to pass-through directly when piping instead of buffering it?

oɔɯǝɹ
  • 7,219
  • 7
  • 58
  • 69
Sebazzz
  • 1,205
  • 2
  • 15
  • 33
  • Try `svnadmin.exe ... | out-string | svnfilter.exe ...`? This may convince powershell to pipe each line instead of collecting everything. I don't have SVN nor know it, so apologies if this breaks horribly. – Eris Dec 22 '14 at 19:07

1 Answers1

5

It's not the output being buffered, but rather the input to any external process. You can verify the behavior with a function like this:

function Read-Pipeline {
[cmdletbinding()]
param ([Parameter(Mandatory = $true, ValueFromPipeline=$true)] $inp)
    Begin {}
    Process {Write-Verbose $inp ; Return $inp}
    End {}
}

If you then run:

.\LongRunning.exe | Read-Pipeline -Verbose | .\Other.exe

You'll see the LongRunning.exe's output in Verbose but Other.exe won't be run until its pipeline is closed. If you do this:

.\LongRunning.exe | Read-Pipeline -Verbose | Write-Host

You'll see alternating Verbose / Console output lines with no buffering because inputs aren't crossing process boundaries.

None of that really helps you. You could work around this by falling back to .NET to launch the processes and manually copying STDOUT to STDIN [1], but it's a lot of work for little reward. The simplest thing to do would be pass your commands to CMD, something like:

& cmd.exe "/C svnadmin.exe dump $FromRepoPath ^| svndumpfilter.exe include --drop-empty-revs --renumber-revs --preserve-revprops $Folder ^| svnadmin.exe load --ignore-uuid $ToRepoPath

[1] http://sushihangover.blogspot.com/2012/01/powershell-piping-standard-output-to.html

tby
  • 351
  • 2
  • 8
  • Thanks, that is pretty much what I feared. What does "|^" do in your example? – Sebazzz Jan 21 '15 at 06:19
  • ^| escapes the | character, which is necessary because it's being passed as a parameter to cmd.exe. http://ss64.com/nt/syntax-esc.html – tby Jan 21 '15 at 11:52
  • 2022 and this still seems to be a problem.. it makes piping large amounts of data kinda impossible. I guess it's back to good old cmd. – stmax Aug 12 '22 at 20:30