Search large .log for specific string quickly without streamreader

Question

Problem: I need to search a large log file that is currently being used by another process. I cannot stop this other process or put a lock on the .log file. I need to quickly search this file, and I can't read it all into memory. I get that StreamReader() is the fastest, but I can't figure out how to avoid it attempting to grab a lock on the file.

$p = "Seachterm:Search"
$files = "\\remoteserver\c\temp\tryingtofigurethisout.log"

$SearchResult= Get-Content -Path $files | Where-Object { $_ -eq $p }

The below doesn't work because I can't get a lock of the file.

$reader = New-Object System.IO.StreamReader($files)

$lines = @()

if ($reader -ne $null) {
    while (!$reader.EndOfStream) {
        $line = $reader.ReadLine()
        if ($line.Contains($p)) {
            $lines += $line
        }
    }
}

$lines | Select-Object -Last 1

This takes too long:

get-content $files -ReadCount 500 |
 foreach { $_ -match $p }

I would greatly appreciate any pointers in how I can go about quickly and efficiently (memory wise) searching a large log file.

Why not make a copy of the file first? If that is not an option you could likely adapt this to suit your needs https://stackoverflow.com/questions/1606349/can-i-prevent-a-streamreader-from-locking-a-text-file-whilst-it-is-in-use by setting the fileopen flag. — Matt, Apr 16 '19 at 19:40
We haven't heard from you about this.. Did you get it resolved the way you want? Didi my answer or the comment by @Matt help you? — Theo, May 01 '19 at 12:59

score 0 · Answer 1 · answered Apr 16 '19 at 20:06

Perhaps this will work for you. It tries to read the lines of the file as fast as possible, but with a difference to your second approach (which is approx. the same as what [System.IO.File]::ReadAllLines() would do).

To collect the lines, I use a List object which will perform faster than appending to an array using +=

$p    = "Seachterm:Search"
$path = "\\remoteserver\c$\temp\tryingtofigurethisout.log"

if (!(Test-Path -Path $path -PathType Leaf)) {
    Write-Warning "File '$path' does not exist"
}
else {
    try {
        $fileStream   = [System.IO.FileStream]::new($path, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::ReadWrite)
        $streamReader = [System.IO.StreamReader]::new($fileStream)

        # or use this syntax:
        # $fileMode     = [System.IO.FileMode]::Open
        # $fileAccess   = [System.IO.FileAccess]::Read
        # $fileShare    = [System.IO.FileShare]::ReadWrite
        # $fileStream   = New-Object -TypeName System.IO.FileStream $path, $fileMode, $fileAccess, $fileShare
        # $streamReader = New-Object -TypeName System.IO.StreamReader -ArgumentList $fileStream

        # use a List object of type String or an ArrayList to collect the strings quickly
        $lines = New-Object System.Collections.Generic.List[string]
        # read the lines as fast as you can and add them to the list
        while (!$streamReader.EndOfStream) {
            $lines.Add($streamReader.ReadLine())
        }
        # close and dispose the obects used
        $streamReader.Close()
        $fileStream.Dispose()

        # do the 'Contains($p)' after reading the file to not slow that part down
        $lines.ToArray() | Where-Object { $_.Contains($p) } | Select-Object -Last 1
    }
    catch [System.IO.IOException] {}
}

Basically, it does what your second code does, but with the difference that using just the StreamReader, the file is opened with [System.IO.FileShare]::Read, whereas this code opens the file with [System.IO.FileShare]::ReadWrite

Note that there may be exceptions thrown using this because another application has write permissions to the file, hence the try{...} catch{...}

Hope that helps

Search large .log for specific string quickly without streamreader

1 Answers1