Loop through files in a directory using PowerShell

Question

How can I change the following code to look at all the .log files in the directory and not just the one file?

I need to loop through all the files and delete all lines that do not contain "step4" or "step9". Currently this will create a new file, but I'm not sure how to use the for each loop here (newbie).

The actual files are named like this: 2013 09 03 00_01_29.log. I'd like the output files to either overwrite them, or to have the SAME name, appended with "out".

$In = "C:\Users\gerhardl\Documents\My Received Files\Test_In.log"
$Out = "C:\Users\gerhardl\Documents\My Received Files\Test_Out.log"
$Files = "C:\Users\gerhardl\Documents\My Received Files\"

Get-Content $In | Where-Object {$_ -match 'step4' -or $_ -match 'step9'} | `
Set-Content $Out

score 486 · Accepted Answer · edited Mar 22 '16 at 13:59

486

Give this a try:

Get-ChildItem "C:\Users\gerhardl\Documents\My Received Files" -Filter *.log | 
Foreach-Object {
    $content = Get-Content $_.FullName

    #filter and save content to the original file
    $content | Where-Object {$_ -match 'step[49]'} | Set-Content $_.FullName

    #filter and save content to a new file 
    $content | Where-Object {$_ -match 'step[49]'} | Set-Content ($_.BaseName + '_out.log')
}

edited Mar 22 '16 at 13:59

Vincent G

361
1
4
22

answered Sep 17 '13 at 11:37

Shay Levy

121,444
32
184
206

thanks. The BaseName part does not work (version 1) - but I managed by first creating a backup of the full file then making the change to the original file. – user2725402 Sep 17 '13 at 12:25
7

For v1, you can use the following to extract the base name: [io.path]::GetFileNameWithoutExtension($name) – Shay Levy Sep 17 '13 at 12:26
1

If I replace `BaseName` with `FullName` I get what I want. – M-- Oct 31 '18 at 13:57

PVitt · Answer 2 · 2019-02-13T20:24:41.393

138

To get the content of a directory you can use

$files = Get-ChildItem "C:\Users\gerhardl\Documents\My Received Files\"

Then you can loop over this variable as well:

for ($i=0; $i -lt $files.Count; $i++) {
    $outfile = $files[$i].FullName + "out" 
    Get-Content $files[$i].FullName | Where-Object { ($_ -match 'step4' -or $_ -match 'step9') } | Set-Content $outfile
}

An even easier way to put this is the foreach loop (thanks to @Soapy and @MarkSchultheiss):

foreach ($f in $files){
    $outfile = $f.FullName + "out" 
    Get-Content $f.FullName | Where-Object { ($_ -match 'step4' -or $_ -match 'step9') } | Set-Content $outfile
}

edited Feb 13 '19 at 20:24

answered Sep 17 '13 at 10:23

PVitt

11,500
5
51
85

This creates one output file named "out.log". How can I create a separate output file for each input file? – user2725402 Sep 17 '13 at 10:31
@user2725402 I added an input dependend output file that is named out – PVitt Sep 17 '13 at 10:39
not sure what I'm doing wrong but this is not working - I now get no output files (and no errors): `$files = Get-ChildItem "C:\Users\gerhardl\Documents\My Received Files\" $files | Where-Object { $outFile = $_.Name + "out" Get-Content $_.FullName | Where-Object { $_ -match 'step4' -or $_ -match 'step9' } | Set-Content $outFile }` – user2725402 Sep 17 '13 at 10:48
@user2725402 I've rewritten the answer using a for loop. – PVitt Sep 17 '13 at 11:24
1

okay 2 new issues... this now for some reason does the opposite of the match, i.e. it creates a file with only records NOT containing "step4" or "step9". Also, it is renaming using "out" as an extension, so files are named "XXX.log.out" - I tried using BaseName but can't get it right. – user2725402 Sep 17 '13 at 12:01
Sorry, I forgot to remove the negation. Use Where-Object { $_ -match 'step4' -or $_ -match 'step9' } instead. – PVitt Sep 17 '13 at 13:01
4

You can also do `foreach ($f in Get-ChildItem "C:\Users\gerhardl\Documents\My Received Files\") { ... }` – Soapy Aug 19 '15 at 12:55
2

or use `ForEach ($f in $files){...}` – Mark Schultheiss May 31 '18 at 13:37

score 44 · Answer 3 · answered Apr 28 '16 at 15:23

44

If you need to loop inside a directory recursively for a particular kind of file, use the below command, which filters all the files of doc file type

$fileNames = Get-ChildItem -Path $scriptPath -Recurse -Include *.doc

If you need to do the filteration on multiple types, use the below command.

$fileNames = Get-ChildItem -Path $scriptPath -Recurse -Include *.doc,*.pdf

Now $fileNames variable act as an array from which you can loop and apply your business logic.

answered Apr 28 '16 at 15:23

Sarath Subramanian

20,027
11
82
86

4

What bugged me was the omission of hidden and/or system files. To also see them you have to use the `-Force` parameter to Get-ChildItem. See its doc at https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-childitem?view=powershell-7 . – til_b Feb 13 '20 at 11:45
Note: The Include parameter only works when either the command includes the Recurse parameter or the path leads to the contents of a directory (https://stackoverflow.com/a/41700792) – Design.Garden Apr 27 '23 at 18:23

user1628658 · Answer 4 · 2015-05-11T08:42:49.453

-18

Other answers are great, I just want to add... a different approach usable in PowerShell: Install GNUWin32 utils and use grep to view the lines / redirect the output to file http://gnuwin32.sourceforge.net/

This overwrites the new file every time:

grep "step[49]" logIn.log > logOut.log

This appends the log output, in case you overwrite the logIn file and want to keep the data:

grep "step[49]" logIn.log >> logOut.log

Note: to be able to use GNUWin32 utils globally you have to add the bin folder to your system path.

edited May 11 '15 at 08:42

answered May 11 '15 at 08:30

user1628658

643
1
6
12

27

This question was tagged as PowerShell, not Unix – SteveC Sep 21 '16 at 15:13
2

Yes. Obviously. My take was fairly simple: You can use GNUWin32 utils in Windows therefore you can use *nix tools in Powershell easily. Works. No one ever said, that the answer had to entirely be native to Powershell. And even if they did, you still can call Win32 Utils directly from Powershell. AFAIK using whatever is available and compatible is not actually a bad thing. – user1628658 Nov 09 '17 at 22:45
3

I would have edited my previous comment, but I passed the 5 minutes limit: SteveC: Are you saying that incorporating GNU tools that work fine under Windows is somehow defying Powershell? That's one thing. I understand that working this out purely in Powershell might be more productive in short term. In long term expanding the possibilities has actually more merit. You add more options and are working entirely in Windows and Powershell without any Linux emulators like Cygwin. So while I understand your reservation, I disagree. – user1628658 Nov 09 '17 at 22:55
8

The question was about looking in PowerShell, not the multitude of other languages that could be used. – SteveC Nov 10 '17 at 07:06
I didn't notice your reply before. Let me reiterate: you don't require to use a different language. GNU Win32Utils work perfectly well from CMD and from PowerShell. Same language, different "include". GNU Win32Utils are meant to be used in Windows. If you won't get it on the 3rd try, I don't think I'll bother again. – user1628658 Dec 13 '19 at 13:55
6

Answers are meant to be to the question. This question was explicit in that it wanted the answer in PowerShell. Your answer was install something and use it. You could have suggested install Ruby, etc. – SteveC Dec 13 '19 at 14:02

Loop through files in a directory using PowerShell

4 Answers4

Linked

Related