Powershell: Get-Content from the large file (server list)

Question

I have 100,000 list of servers from the text file (serverlist.txt)

When I run in one shot it will burst my memory and cpu and the time took longer (about 3 days)to complete the scanning for DNSlookup.

I tried to split the file that contain 20k list of servers below and can be completed to scan up to 10mins for each file.

serverlist1.txt
serverlist2.txt
serverlist3.txt
serverlist4.txt
serverlist5.txt

$objContainer = @()
$values = @()
$domains = Get-Content -path "serverlist1.txt"
$named = 0
$timestamp= get-date

$domains | ForEach-Object {
    $domain = $_
    nslookup $domain 2>&1 | ForEach-Object {
        if ($_ -match '^Name:\s*(.*)$') {
            $values += $matches[1]
            $named = 1;
        } elseif (($_ -match '^.*?(\d*\.\d*\.\d*\.\d*)$') -and ($named -eq 1)) {
            $values += $matches[1]
        } elseif ($_ -match '^Aliases:\s*(.*)$') {
            $values += $matches[1]
        }
    }

    $obj = New-Object -TypeName PSObject
    #$obj | Add-Member -MemberType NoteProperty -name 'Domain' -value $domain
    $obj | Add-Member -MemberType NoteProperty -name 'Name' -value $values[0]
    $obj | Add-Member -MemberType NoteProperty -name 'IP Address' -value $values[1]
    $obj | Add-Member -MemberType NoteProperty -name 'Alias' -value $values[2]
    $obj | Add-Member -MemberType NoteProperty -name 'Timestamp' -value $timestamp
    $objContainer += $obj

    $values = @()
    $named = 0
}

Write-Output $objContainer
$objContainer | Export-csv "dnslog_$((Get-Date).ToString('MM-dd-yyyy_hh-mm-ss')).csv" -NoTypeInformation

My question is, how to execute at once and looping the input from the text file after generate the dnslog(datetime).csv

e.g:

run the powershell script .\filename.ps1
input from serverlist1.txt
output dnslog(datetime).csv
input from serverlist2.txt
output dnslog(datetime).csv
input from serverlist3.txt
output dnslog(datetime).csv
input from serverlist4.txt
output dnslog(datetime).csv
input from serverlist5.txt
output dnslog(datetime).csv

Finish!

If i have more then 5 list of text file, it will continue to loop from the input file until completed.

score 0 · Answer 1 · answered Aug 12 '16 at 10:05

You should consider running this a parallel batching job. Have you already tried doing so?

You can deal with the RAM busting problem by removing all those commits to memory (variable assignments and array rewriting with +=).

$timestamp = get-date

Get-Content -path "serverlist1.txt" | ForEach-Object {
    $domain = $_

    # You can clear this here.
    $values = @()
    $named = 0

    # There are potentially better options than nslookup.
    # Needs a bit of care to understand what's an alias here though.
    # [System.Net.Dns]::GetHostEntry($domain)
    # And if you don't like that, quite a few of us have written equivalent tools in PowerShell.
    nslookup $domain 2>&1 | ForEach-Object {
        if ($_ -match '^Name:\s*(.*)$') {
            $values += $matches[1]
            $named = 1;
        } elseif (($_ -match '^.*?(\d*\.\d*\.\d*\.\d*)$') -and ($named -eq 1)) {
            $values += $matches[1]
        } elseif ($_ -match '^Aliases:\s*(.*)$') {
            $values += $matches[1]
        }
    }

    # Leave the output object in the output pipeline
    # If you're running PowerShell 3 or better:
    [PSCustomObject]@{
        Domain       = $domain
        Name         = $values[0]
        'IP Address' = $values[1]
        Alias        = $values[2]
        TimeStamp    = $timestamp
    }
    # PowerShell 2 is less flexible. This or Select-Object.
    #$obj = New-Object -TypeName PSObject
    ##$obj | Add-Member -MemberType NoteProperty -name 'Domain' -value $domain
    #$obj | Add-Member -MemberType NoteProperty -name 'Name' -value $values[0]
    #$obj | Add-Member -MemberType NoteProperty -name 'IP Address' -value $values[1]
    #$obj | Add-Member -MemberType NoteProperty -name 'Alias' -value $values[2]
    #$obj | Add-Member -MemberType NoteProperty -name 'Timestamp' -value $timestamp
    # To leave this in the output pipeline, uncomment this
    # $obj

    # No version of PowerShell needs you to do this. It's a good way to ramp up memory usage 
    # for large data sets.
    # $objContainer += $obj
} | Export-Csv "dnslog_$(Get-Date -Format 'MM-dd-yyyy_hh-mm-ss').csv" -NoTypeInformation

score 0 · Accepted Answer · answered Aug 12 '16 at 13:36

0

Adding to Chris's answer I would also add a ReadCount flag to the Get-Content like so:

Get-Content -path "serverlist1.txt" -ReadCount 1 | % {

This will save having to read the entire file into memory.

answered Aug 12 '16 at 13:36

Dave Sexton

10,768
3
42
56

Powershell: Get-Content from the large file (server list)

2 Answers2