Powershell Job Memory Consumption Issue

Question

I've been struggling with this for a week now and have exhausted all the methods and options I have found online. I am hoping someone here will be able to help me out with this.

I am using powershell to start 8 jobs, each job running FFmpeg to stream a 7 minute file to a remote RTMP server. This is pulling from a file on the disk and each job uses a different file. The command is in a do while loop so that it is constantly restreaming.

This is causing the shell I launched the jobs from to accumulate a massive amount of memory, consuming all that it can. In 24 hours it consumed 30 of the 32 GB of my server.

Here is my launch code, any help would be appreciated.

start-job -Name v6 -scriptblock {
do { $d = $true; $f = Invoke-Expression -Command "ffmpeg -re -i `"C:\Shares\Matthew\180p_3000k.mp4`" -vcodec copy -acodec copy -f flv -y rtmp://<ip>/<appName>/<streamName>"; $f = $null }
while ($d = $true)

}

I've tried to receive the jobs and pipe it to out-null, I've tried setting $f to $null before starting the do while loop, and some other things I found online but to no avail. Thanks everyone for your time!

**do { } while ($d = $true)** is an infinite loop. Your expression will be invoked over and over and over ....If you want to test for condition do **while ($d -eq $true)** Then again, 'do { $d = $true; } while ($d -eq $true)' is also an infinite loop unless you set $d to $false somewhere. Something wrong with your logic there. Can you try to put the logic into words in your question? Constantly re-streaming? Or re-streaming when one stream ends? — Jan Chrbolka, Mar 31 '15 at 01:44
Oh man a stupid mistake I made again.. I will change it to $d -eq $true but it is supposed to be an infinite looo. When the 7 minute clip ends it should just re stream it again. Maybe I should be not making an infinite loop and waiting for the job to end and recast the job? — Matt Wall, Mar 31 '15 at 19:45

score 4 · Accepted Answer · edited May 23 '17 at 12:32

4

Better late than never I guess. I've had the same problem with huge memory consumption when running ffmpeg in Powershell jobs. The core of the issue is that a Powershell job will store any/all output into memory, and ffmpeg is extremely happy to log output to both standard output and standard error streams.

My solution was to add the parameter "-loglevel quiet" to ffmpeg. Alternatively you could redirect both the standard and error streams to null (it's not enough to redirect just the standard stream). For more on how to redirect the standard streams, refer to this question: Redirection of standard and error output appending to the same log-file

edited May 23 '17 at 12:32

Community

1
1

answered Jul 14 '16 at 13:53

Helge Elvik

56
2

Thanks that was useful! I'm going to try that. The work around I had for some time was to stuff the ffmpeg commands into batch files. If I needed to be dynamic about it I would write the commands as a batch file and run them from the script. It seemed this way memory did not creep up. That further proves your point - I believe what I was doing was causing a degree of separation to prevent any stdout or stderr from staying in the powershell namespace. – Matt Wall Aug 07 '16 at 07:48

score 2 · Answer 2 · answered Apr 01 '15 at 00:49

There are many ways to invoke a PowerShell script or an expression.

One of my favorites is using RunspacePool. In a RunspacePool each task is started with a new runspace. When the task completes the runspace is disposed of. This should keep memory consumption fairly constant. It is not the easiest method to get your head around, but here is a bare-bone example that might work for you.

$Throttle = 10 # Maximum jobs that can run simultaneously in a runpool
$maxjobs = 8 # Maximum jobs that will run simultaneously
[System.Collections.ArrayList]$jobs = @()
$script:currentjobs = 0
$jobindex = 0
$jobaction = "ffmpeg -re -i `"C:\Shares\Matthew\180p_3000k.mp4`" -vcodec copy -acodec copy -f flv -y rtmp://<ip>/<appName>/<streamName>"

$sessionstate = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
$runspace = [runspacefactory]::CreateRunspacePool(1, $Throttle, $sessionstate, $Host)
$runspace.Open()


function QueueJob{
    param($jobindex)
    $job = "" | Select-Object JobID,Script,Errors,PS,Runspace,Handle
    $job.JobID = $jobindex
    $job.Script = $jobaction
    $job.PS = [System.Management.Automation.PowerShell]::create()
    $job.PS.RunspacePool = $runspace
    [void]$job.PS.AddScript($job.Script)
    $job.Runspace = $job.PS.BeginInvoke()
    $job.Handle = $job.Runspace.AsyncWaitHandle
    Write-Host "----------------- Starting Job: $("{0:D4}" -f $job.JobID) --------------------" -ForegroundColor Blue
    $jobs.add($job)
    $script:currentjobs++
}

Function ServiceJobs{
    foreach($job in $Jobs){
        If ($job.Runspace.isCompleted) {
            $result = $job.PS.EndInvoke($job.Runspace)
            $job.PS.dispose()
            $job.Runspace = $null
            $job.PS = $null
            $script:currentjobs--
            Write-Host "----------------- Job Completed: $("{0:D4}" -f $job.JobID) --------------------" -ForegroundColor Yellow
            Write-Host $result
            $Jobs.Remove($job)
            return
        }
    }
}

while ($true) {
    While ($script:currentjobs -le $maxjobs){
        QueueJob $jobindex
        $jobindex++
    }
    ServiceJobs
    Start-Sleep 1
}

$runspace.Close()
[gc]::Collect()

It is using an infinite loop without proper termination and is lacking any sort of error checking, but hopefully it is enough to demonstrate the technique.

I just saw this - thank you - I'm going to try to figure out how to use this. I love the idea and I think my actual solution to this problem back when was sort of indirectly launching ffmpeg in a new namespace by running it via invoking batch files written dynamically. — Matt Wall, Aug 07 '16 at 07:49

Powershell Job Memory Consumption Issue

2 Answers2