How to iterate a text file for a specific key and save in variable using powershell?

Question

I am trying to create a script which will loop through a text file <-- (Step 3.1) and find for specific keys and saving the values to a variable. Now that variable I am using in a command to generate the output file < --(step 4). I am using the code as of now.

Code:

# 1- Connect to Azure Account

$username = "abc@xyz.com"
$pass = ConvertTo-SecureString "abc123" -AsPlainText -Force
$cred = New-Object PSCredential($username,$pass)

#Connect-AzureRmAccount -Credential $cred | out-null
Connect-AzAccount -Credential $cred | out-null

# 2 - Input Area

$subscriptionName = 'Data Analytics'
$resourceGroupName = 'DataLake-Gen2'
 $dataFactoryName = 'dna-production-gen2'
$runStartDateTimeUTC = '2020-09-12T06:40:00Z'

# 3 - (All Triggers Information)

$ErrorActionPreference="SilentlyContinue"
Stop-Transcript | out-null
$ErrorActionPreference = "Continue"
Start-Transcript -path D:\Powershell\TriggerInfo.txt -append
Get-AzDataFactoryV2Trigger -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName
Stop-Transcript  

# 3.1 (Get the content in a variable)

Get-Content "D:\Powershell\TriggerInfo.txt" | ForEach-Object {
$trg_name = $_.TriggerName
} $trg_name

# 4 - (Write the output to a text file, The Get-AzDataFactoryV2TriggerRun command returns detailed information about trigger runs for the specified trigger 
#  in the given timeframe.)

foreach ($trg in $trg_name) {

$ErrorActionPreference="SilentlyContinue"
Stop-Transcript | out-null
$ErrorActionPreference = "Continue"
Start-Transcript -path D:\Powershell\output.txt -append
Get-AzDataFactoryV2TriggerRun -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -TriggerName $trg -TriggerRunStartedAfter "2020-09-01" -TriggerRunStartedBefore "2020-09-17"
Stop-Transcript

"$trg = " + $trg.length
}

TriggerInfo.txt:

TriggerName       : TRG_CM_TBLEnhanced_prod
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CM_tblEnhanced_QA
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CustCaseData
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_RP_Dashboard_TAE
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

Any help will be appreciated? Thanks

Updated Code

    # 3.1 (Get the content in a variable)
    
    # Get-Content "D:\Powershell\TriggerInfo.txt" | ForEach-Object {
    # $trg_name = $_.TriggerName
    # } $trg_name
    
    # $trg = (Get-AzDataFactoryV2Trigger -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName).TriggerName
    
    $log = Get-Content -Path 'D:\Powershell\TriggerInfo.txt' -Raw
    $result = ($log -split '(\r?\n){2,}' | Where-Object {$_ -match '\S'}) | ForEach-Object {
    [PsCustomObject](($_ -split 'TriggeredPipelines')[0] -replace ' : ', '=' | ConvertFrom-StringData)  |
        Select-Object 'TriggerName'
    }
    
    # output on screen
    $result | Format-Table -AutoSize
    
    # write to CSV file
    $result | Export-Csv -Path 'D:\Powershell\TriggerInforesult.csv' -Encoding UTF8 -NoTypeInformation -Force
    
    $trg_name = (Import-Csv -Path 'D:\Powershell\TriggerInforesult.csv').TriggerName
    
    # 4 - (Write the output to a text file, The Get-AzDataFactoryV2TriggerRun command returns detailed information about trigger runs for the specified trigger 
    #  in the given timeframe.)
    
    foreach ($trigger in $trg_name) {
    
    $ErrorActionPreference="SilentlyContinue"
    Stop-Transcript | out-null
    $ErrorActionPreference = "Continue"
    Start-Transcript -path D:\Powershell\output.txt -append
    
    Get-AzDataFactoryV2TriggerRun -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -TriggerName $trigger -TriggerRunStartedAfter "2020-09-01" -TriggerRunStartedBefore "2020-09-17"
    
    Stop-Transcript
    
    "$trigger = " + $trigger.length
    }

****Recent Update****

TriggerName       : TRG_CM_TBLEnhanced_prod
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CM_tblEnhanced_QA
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CustCaseData
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_RP_Dashboard_TAE
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : GMB_Trigger
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.BlobEventsTrigger
RuntimeState      : Started

TriggerName       : TRG_RP_Optimizely_Import
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_USBusinessData_Monthly
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_Generic_CSV_To_DW
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Stopped

TriggerName       : TRG_CM_Dimension_Unit
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CM_PricingQuoteApproval
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

Transcript stopped, output file is D:\Powershell\new\TriggerInfo.txt
ConvertFrom-StringData : Data line '**********************' is not in 'name=value' format.
At line:36 char:37
+     $data  = $_ -replace ':', '=' | ConvertFrom-StringData
+                                     ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [ConvertFrom-StringData], PSInvalidOperationException
    + FullyQualifiedErrorId : InvalidOperation,Microsoft.PowerShell.Commands.ConvertFromStringDataCommand

Get-AzDataFactoryV2TriggerRun : Cannot validate argument on parameter 'DataFactoryName'. The argument is null or empty. Provide an argument that is not null or empty, and then try the
command again.
At line:47 char:34
+    Get-AzDataFactoryV2TriggerRun @splat
+                                  ~~~~~~
    + CategoryInfo          : InvalidData: (:) [Get-AzDataFactoryV2TriggerRun], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.Azure.Commands.DataFactoryV2.GetAzureDataFactoryTriggerRunCommand

ConvertFrom-StringData : Data line '**********************' is not in 'name=value' format.
At line:36 char:37
+     $data  = $_ -replace ':', '=' | ConvertFrom-StringData
+                                     ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [ConvertFrom-StringData], PSInvalidOperationException
    + FullyQualifiedErrorId : InvalidOperation,Microsoft.PowerShell.Commands.ConvertFromStringDataCommand

TriggerInfo.txt

Windows PowerShell transcript start Start time: 20201017093947 Username: XXXXXX\XXXXXX RunAs User: XXXXXX\XXXXXX Configuration Name: Machine: INNOPHLTXETR138 (Microsoft Windows NT 10.0.17134.0) Host Application: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe Process ID: 20340 PSVersion: 5.1.17134.858 PSEdition: Desktop PSCompatibleVersions: 1.0, 2.0, 3.0, 4.0, 5.0, 5.1.17134.858 BuildVersion: 10.0.17134.858 CLRVersion: 4.0.30319.42000 WSManStackVersion: 3.0 PSRemotingProtocolVersion: 2.3 SerializationVersion: 1.1.0.1

Transcript started, output file is D:\Powershell\new\TriggerInfo.txt

TriggerName       : TRG_CM_TBLEnhanced_prod
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CM_tblEnhanced_QA
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CustCaseData
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_RP_Dashboard_TAE
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : GMB_Trigger
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.BlobEventsTrigger
RuntimeState      : Started

TriggerName       : TRG_RP_Optimizely_Import
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_USBusinessData_Monthly
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_Generic_CSV_To_DW
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Stopped

TriggerName       : TRG_CM_Dimension_Unit
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CM_PricingQuoteApproval
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

**********************
Windows PowerShell transcript end
End time: 20201017093951
**********************

You really making it yourself very difficult, this is not how PowerShell is supposed to work. You might simply grap your triggers from the [Az.DataFactory](https://learn.microsoft.com/en-us/powershell/module/az.datafactory/get-azdatafactoryv2trigger?view=azps-4.8.0) object like this: `(Get-AzDataFactoryV2Trigger -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName).TriggerName` — iRon, Oct 16 '20 at 08:31
Hi iRon, Thanks for the suggestion. I am actually newbie to Powershell. Triggers info. I have already saved in the triggerinfo file, now I need my output file generated for each individual triggername and append the o/p to output file. — Saurabh, Oct 16 '20 at 08:51
hmm, this really not the way to go. Anyways, this [`ConvertFrom-SourceTable`](https://www.powershellgallery.com/packages/ConvertFrom-SourceTable) cmdlet might help you (see also: [Parsing PowerShell space-separated output as a table](https://stackoverflow.com/a/51213180/1701026). If you add (a part) of the `TriggerInfo.txt` output (to the question), I might be able to be more specific. — iRon, Oct 16 '20 at 09:06
Thanks iRon for update, also I have updated the question with sample TriggerInfo. — Saurabh, Oct 16 '20 at 09:31
The variable that will hold the trigger names, will be passed as an parameter to the next command. I am not sure how to do this ? — Saurabh, Oct 16 '20 at 09:48
As for your Step 3.1. `Get-Content` reads a file into a string array. From that you would need to parse it out in various properties. However, from your [earlier question](https://stackoverflow.com/questions/64280025/extracting-the-key-value-pair-in-loop-from-text-file-using-powershell-script), you have all you need already in a **structured** CSV file, so why don't you use that like this: `$trg_name = (Import-Csv -Path 'TheResultsFileFromEarlierQuestion.CSV').TriggerName`. Then `$trg_name` will contain the array of TriggerNames you want. — Theo, Oct 16 '20 at 09:54
@theo: Thanks for the suggestion, Now I am using the same like you described but when I am trying to fetch over the array and passing out to the parameter, it is showing the output to the console but not writing to the output.txt. — Saurabh, Oct 16 '20 at 10:21
Hi Theo, I am sorry for the same. Actually I have clicked on every answer that solved my issue, but because I do have low reputation score being a newbie here. My Vote doesn't count. Hope you understand. It always shows me that you have low reputation count, BTW I have already accepted the answer as solution earlier. — Saurabh, Oct 22 '20 at 12:41
Thanks for accepting. You have just gained +2 reputation for that! — Theo, Oct 22 '20 at 12:46

Theo · Accepted Answer · 2020-10-17T14:08:36.473

As commented, Get-Content reads a file as string array. If you want to use the example textfile you gave in the question, you need parse it out as an array of objects in order to select the property or properties you want filtered out.

However, in your previous question, you have already done the hard work and have a structured CSV file with all the properties you'll ever want, so I suggest you use that for input.

The code can then be as simple as:

$trg_name = (Import-Csv -Path 'TheResultsFileFromEarlierQuestion.CSV').TriggerName

and use that to save the $trigger array to a file, or use it for iterating stuff:

$trg_name | ForEach-Object {
    # using splatting here for prettier code
    $splat = @{
        ResourceGroupName       = $resourceGroupName
        DataFactoryName         = $dataFactoryName
        TriggerName             = $_
        TriggerRunStartedAfter  = "2020-09-01"
        TriggerRunStartedBefore = "2020-09-17"
    }
    Get-AzDataFactoryV2TriggerRun @splat
} | Export-Csv -Path 'PathToTheOutputFile.csv' -Encoding UTF8 -NoTypeInformation

P.S. If $resourceGroupName and $dataFactoryName should also come from the input CSV file, then you don't need $trg_name at all and can simply do:

Import-Csv -Path 'TheResultsFileFromEarlierQuestion.CSV' | ForEach-Object {
    $splat = @{
        ResourceGroupName       = $_.ResourceGroupName
        DataFactoryName         = $_.DataFactoryName
        TriggerName             = $_.TriggerName
        TriggerRunStartedAfter  = "2020-09-01"
        TriggerRunStartedBefore = "2020-09-17"
    }
    Get-AzDataFactoryV2TriggerRun @splat
} | Export-Csv -Path 'PathToTheOutputFile.csv' -Encoding UTF8 -NoTypeInformation

and that then would be the complete code for both Step 3.1 and Step 4

Edit

From your comments, I understand that you can not use the csv file you made earlier for this, and that you're stuck with the input textfile as you show us.

That too is not very hard to use:

# split the text in 'trigger' text blocks on the empty line
$triggers = ((Get-Content "D:\Powershell\TriggerInfo.txt" -Raw) -split '\*+')[0] -split '(\r?\n){2,}'
# loop through these blocks (skip any possible empty textblock)
$triggers | Where-Object {$_ -match '\S'} | ForEach-Object {
    # and parse the data into Hashtables
    $data  = $_ -replace ':', '=' | ConvertFrom-StringData
    # if you have values containing a colon (:) character, this is safer:
    # $data  = ($_ -split ':', 2) -join '=' | ConvertFrom-StringData

    $splat = @{ 
        ResourceGroupName       = $data.ResourceGroupName
        DataFactoryName         = $data.DataFactoryName
        TriggerName             = $data.TriggerName
        TriggerRunStartedAfter  = "2020-10-15"
        TriggerRunStartedBefore = "2020-10-17"
   } 
   Get-AzDataFactoryV2TriggerRun @splat 
} | Export-Csv -Path 'D:\Powershell\new\Output.csv' -Encoding UTF8 -NoTypeInformation

Hi Theo, This is the error I am getting "Get-AzDataFactoryV2TriggerRun : Cannot validate argument on parameter 'Name'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again." I can't use the file output from previous question, because that output was only one trigger name, I want to generate the output for each triggername and append it in the output file. That is why I am using the triggerinfo file to store and loop through the name of the trigger and pass it as a parameter to AzDataFactoryV2TriggerRun. Hope you get it. — Saurabh, Oct 17 '20 at 11:24
$trg_name = (Import-Csv -Path 'D:\Powershell\new\TriggerInfo.csv').TriggerName $trg_name | ForEach-Object { # using splatting here for prettier code $splat = @{ ResourceGroupName = $resourceGroupName DataFactoryName = $dataFactoryName TriggerName = $_ TriggerRunStartedAfter = "2020-10-15" TriggerRunStartedBefore = "2020-10-17" } Get-AzDataFactoryV2TriggerRun @splat } | Export-Csv -Path 'D:\Powershell\new\Output.csv' -Encoding UTF8 -NoTypeInformation — Saurabh, Oct 17 '20 at 11:31
@SaurabhShakyawar Please see the last edit. The code now uses the textfile and converts that to usable data to use. — Theo, Oct 17 '20 at 13:25
Error 1: ConvertFrom-StringData : Data line '**********************' is not in 'name=value' format. At line:36 char:37 Error 2: Get-AzDataFactoryV2TriggerRun : Cannot validate argument on parameter 'DataFactoryName'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again. At line:47 char:34........................................... I am getting this error — Saurabh, Oct 17 '20 at 13:44
@SaurabhShakyawar Then.... **WHY** does your example input text not show us those lines? Honestly.. If you keep information like that behind, it is shear impossible to answer your question. — Theo, Oct 17 '20 at 13:46
Theo, I have updated now the sample input and error in the Edit --> Recent Update. I apologize for the issue. — Saurabh, Oct 17 '20 at 13:52
@SaurabhShakyawar You should have updated the input text file so it is visible what ugly lines it contains. Updated the answer — Theo, Oct 17 '20 at 13:56
Thanks for the patience Theo. I have updated my TriggerInfo.txt. — Saurabh, Oct 17 '20 at 14:04

How to iterate a text file for a specific key and save in variable using powershell?

1 Answers1