powershell read and write a large data to text file faster

Question

$m_point = Select-String -pattern  ",T," c:\temp\test\mt_point.txt
For ($x=0; $x -le $m_point.length; $x++)
#For ($x=0; $x -le 2; $x++)

{
    $s_point = $m_point[$x] -split ","
   $s_point = $s_point -replace ''''
  # echo $s_point[10]
   $point_ID=$s_point[2]+"."+$s_point[3]+"."+$s_point[4]+"."+$s_point[5]
   # echo $point_ID
        $Sel = Select-String  -pattern $point_ID -path $Location 
        $tp_point_stirng= $Sel-split ";"
      # echo $tp_point_stirng[6]
       # echo $tp_point_stirng[7]
        $tp_point_no=$tp_point_stirng[0]-split ":"
        $tp_point=$tp_point_no[3]-split ","
        #echo $tp_point[1]
   if($point_ID  -eq $tp_point_stirng[1])
   {
    if($s_point[6]-eq "T")
    {
        $Nis_point="!1!"+$tp_point[1]+"!"+$s_point[2]+ "!"+$tp_point_stirng[7]+"!0!SI!!"+$s_point[7]+"!!!!"+ $s_point[8]+"!"
        $Nis_point | out-file -filepath $fname_tag -Append 

    }
     if($s_point[9]-eq "T")
    {
        $Nis_point="!1!"+$tp_point[1]+"!"+$s_point[2]+ "!"+$tp_point_stirng[7]+"!2!AI!!"+$s_point[10]+"!!!!"+ $s_point[11]+"!"
        $Nis_point | out-file -filepath $fname_tag -Append 
    }
    if($s_point[12] -eq "T" -or $s_point[13] -eq "T")
    {
        if($s_point[14] -eq "F" -or $s_point[15] -eq "F")
        {
           $SDID_XDIS= "1"+","+$tp_point[1]+",0"
           $SDID_XDIS | out-file -filepath $fname_mu -Append 
        }
        if($s_point[14] -eq "T" -or $s_point[15] -eq "F")
        {
           $SDID_XDIS="1"+","+$tp_point[1]+",1"
           $SDID_XDIS | out-file -filepath $fname_mu -Append 
        }
        if($s_point[14] -eq "F" -or $s_point[15] -eq "F")
        {
           $SDID_XDIS= "1"+","+$tp_point[1]+",2"
           $SDID_XDIS | out-file -filepath $fname_mu -Append 
        }
        if($s_point[14] -eq "T" -or $s_point[15] -eq "T")
        {
           $SDID_XDIS= "1"+","+$tp_point[1]+",3"
           $SDID_XDIS | out-file -filepath $fname_mu -Append 
        }
    } 
    }
}

first the program read the about 80000 line data,split the line.

split object, search the file,and then split object,

search file use: $Sel = Select-String -pattern $point_ID -path $Location

write file: $SDID_XDIS | out-file -filepath $fname_mu -Append

finish the write text file about 1 hours,very slower.

how can i do? any other faster finish the program

I've got a search-text file function [here](http://stackoverflow.com/a/24091871/1751302). Not perfect, but [system.io.file]::ReadLines($fullPath) is much faster with large files. For [writing](http://stackoverflow.com/a/13855829/1751302) you could try System.IO.StreamWriter($fullPath). — noam, Oct 23 '14 at 10:35
I see lots of string concatenations going on there (very slow). Depending on how many matched records you're getting out of the file, you may be getting slowed down by memory management if $m_results is a large collection. Switching to Get-Content with a -ReadCount of about 1000, and then using -Match on the resulting arrays in a pipeline with foreach may be quicker. — mjolinor, Oct 23 '14 at 10:54

score 0 · Answer 1 · answered Oct 23 '14 at 10:48

Use Measure-Command to find which statements take the most time.

That being said, the code contains two obvious bottlenecks. Reading a large file with Select-String is going to perform poorly. Consider using ReadLine() from System.IO.File instead. Writing to a file row-by-row with Add-Content is another a performance killer. Instead, use buffered output with, say, StringBuilder from System.Text.

The parsing routine for $SDID_XDIS is likely to need a rewrite. Consider that you are checking the same condition multiple times:

    if($s_point[14] -eq "F" -or $s_point[15] -eq "F") # Match F_ or _F
    {
       ...
    }
    if($s_point[14] -eq "T" -or $s_point[15] -eq "F") # Match T_ or _F
    {
       ... # Huh? _F was already checked in the previous step?
    }
    if($s_point[14] -eq "F" -or $s_point[15] -eq "F") # Match F_ or _F
    {
       ... # Huh? Didn't we check this already?           
    }
    if($s_point[14] -eq "T" -or $s_point[15] -eq "T") # Match T_ or _T
    {
       ...
    }

powershell read and write a large data to text file faster

1 Answers1