1

I'm trying to avoid the extremely verbose hash maps and arrays, as commonly used in powershell. Why? Because I have 100's of lines, and it just doesn't make any sense to have to wrap every single line in a @(name='foo; id='bar') etc.), when all I need is a CSV type of array.

$header = @('name', 'id', 'type', 'loc')

$mycsv = @(
    # name, id, type, loc
    'Brave', 'Brave.Brave', 1, 'winget'
    'Adobe Acrobat (64-bit)', '{AC76BA86-1033-1033-7760-BC15014EA700}', 2, ''
    'GitHub CLI', 'GitHub.cli', 3, 'C:\portable'
)

# Do some magic here to set the CSV / hash headers so I can use them as shown below

Foreach ($app in $mycsv) {
    Write-Host "App Name: $app.name"
    Write-Host "App Type: $app.type"
    Write-Host "App id  : $app.id"
    Write-Host "App Loc : $app.type"
    Write-Host ("-"*40)
}

I'm sure you see where I am going.

So how can I process the inline CSV line-by-line using the header names?

Expected output:

App Name: Brave
App Type: 1
App id  : Brave.Brave
App Loc : winget
----------------------------------------
...

UPDATE: 2022-12-03

The ultimate solution is the following very brief and non-verbose code:

$my = @'
    name,id,type,loc
    Brave, Brave.Brave,1,winget
    "Adobe Acrobat (64-bit)",{AC76BA86-1033-1033-7760-BC15014EA700},2,
    GitHub CLI,GitHub.cli,,C:\portable
'@ 

ConvertFrom-Csv $my | % {
    Write-Host "App Name: $($_.name)"
    Write-Host "App Type: $($_.type)"
    Write-Host "App id  : $($_.id)"
    Write-Host "App Loc : $($_.loc)"
    Write-Host $("-"*40)
}

not2qubit
  • 14,531
  • 8
  • 95
  • 135
  • It should be noted that if you need to add to the *here-is* list, you need to add a new-line if using the `+=` method. So that we have `$my += @' Discord, Discord.Discord, 4, zzzz'@`, or: `$my += "\`n Discord, Discord.Discord, 4, zzzz"`. – not2qubit Dec 04 '22 at 01:59

2 Answers2

2

You can use an in-memory, i.e. string representation of CSV data using a here-string and parse it into objects with ConvertFrom-Csv:

# This creates objects ([pscustomobject] instances) with properties
# named for the fields in the header line (the first line), i.e: 
#  .name, .id. .type, and .loc
# NOTE: 
# * The whitespace around the fields is purely for *readability*.
# * If any field values contain "," themselves, enclose them in "..."
$mycsv =
@'
  name,                   id,                                       type, loc
  Brave,                  Brave.Brave,                              1,    winget
  Adobe Acrobat (64-bit), {AC76BA86-1033-1033-7760-BC15014EA700},   2,
  GitHub CLI,             GitHub.cli,                               3,    C:\portable
'@ | ConvertFrom-Csv

$mycsv | Format-List then provides the desired output (without Format-List, you'd get implicit Format-Table formatting, because the objects have no more than 4 properties).

  • As an aside: Format-List in essence provides the for-display formatting you've attempted with your loop of Write-Host calls; if you really need the latter approach, note that, as pointed out in Walter Mitty's answer, you need to enclose property-access expressions such as $_.name in $(...) in order to expand as such inside an expandable (double-quoted) PowerShell string ("...") - see this answer for a systematic overview of the syntax of PowerShell's expandable strings (string interpolation).

Note:

  • This approach is convenient:

    • It allows you to omit quoting, unless needed, namely only if a field value happens to contain , itself.

      • Use "..." (double-quoting) around field values that themselves contain , ('...', i.e. single-quoting does not have syntactic meaning in CSV data, and any ' characters are retained verbatim).

        • Should such a field additionally contain " chars., escape them as ""
    • It allows you to use incidental whitespace for more readable formatting, as shown above.

  • You may also use a separator other than , (e.g., |) in the input and pass it to ConvertFrom-Csv via the -Delimiter parameter.

  • Note: CSV data is in general untyped, which means that ConvertFrom-Csv (as well as Import-Csv) creates objects whose properties are all strings ([string]-typed).


Optional reading: A custom CSV notation that enables creation of typed properties:

Convenience function ConvertFrom-CsvTyped (source code below) overcomes the limitation of ConvertFrom-Csv invariably creating only string-typed properties, by enabling a custom header notation that supports preceding each column name in the header line with a type literal; e.g. [int] ID (see this answer for a systematic overview of PowerShell's type literals, which can refer to any .NET type).

This enables you to create (non-string) typed properties from the input CSV, as long as the target type's values can be represented as numbers or string literals, which includes:

  • Numeric types ([int], [long], [double], [decimal], ...)
  • Date and time-related types [datetime], [datetimeoffset], and [timespan]
  • [bool] (use 0 and 1 as the column values)
  • To test whether a given type can be used, cast it from a sample number or string, e.g.: [timespan] '01:00' or [byte] 0x40

Examples - note the type literals preceding the 2nd and third column names, [int] and [datetime] :

@'
  Name,        [int] ID, [datetime] Timestamp
  Forty-two,   0x2a,     1970-01-01
  Forty-three, 0x2b,     1970-01-02
'@ | ConvertFrom-CsvTyped

Output - note how the hex. numbers were recognized as such (and formatted as decimals by default), and how the data strings were recognized as [datetime] instances:

Name        ID Timestamp
----        -- ---------
Forty-two   42 1/1/1970 12:00:00 AM
Forty-three 43 1/2/1970 12:00:00 AM

Adding -AsSourceCode to the call above allows you to output the parsed objects as a PowerShell source code string, namely as an array of [pscustomobject] literals:

@'
  Name,        [int] ID, [datetime] Timestamp
  Forty-two,   0x2a,     1970-01-01
  Forty-three, 0x2b,     1970-01-02
'@ | ConvertFrom-CsvTyped -AsSourceCode

Output - note that if you were to use this in a script or as input to Invoke-Expression (for testing only), you'd get the same objects and for-display output as above:

@(
  [pscustomobject] @{ Name = 'Forty-two'; ID = [int] 0x2a; Timestamp = [datetime] '1970-01-01' }
  [pscustomobject] @{ Name = 'Forty-three'; ID = [int] 0x2b; Timestamp = [datetime] '1970-01-02' }
)

ConvertFrom-CsvTyped source code:
function ConvertFrom-CsvTyped {
  <#
.SYNOPSIS
  Converts CSV data to objects with typed properties;
.DESCRIPTION
  This command enhances ConvertFrom-Csv as follows:
   * Header fields (column names) may be preceded by type literals in order
     to specify a type for the properties of the resulting objects, e.g. "[int] Id"
   * With -AsSourceCode, the data can be transformed to an array of 
    [pscustomobject] literals.

.PARAMETER Delimiter
  The single-character delimiter (separator) that separates the column values.
  "," is  the (culture-invariant) default.

.PARAMETER AsSourceCode
  Instead of outputting the parsed CSV data as objects, output them as
  as source-code representations in the form of an array of [pscustomobject] literals.

.EXAMPLE
  "Name, [int] ID, [datetime] Timestamp`nForty-two, 0x40, 1970-01-01Z" | ConvertFrom-CsvTyped
  
  Parses the CSV input into an object with typed properties, resulting in the following for-display output:
    Name      ID Timestamp
    ----      -- ---------
    Forty-two 64 12/31/1969 7:00:00 PM  

  .EXAMPLE
  "Name, [int] ID, [datetime] Timestamp`nForty-two, 0x40, 1970-01-01Z" | ConvertFrom-CsvTyped -AsSourceCode
  
  Transforms the CSV input into an equivalent source-code representation, expressed
  as an array of [pscustomobject] literals:
    @(
      [pscustomobject] @{ Name = 'Forty-two'; ID = [int] 0x40; Timestamp = [datetime] '1970-01-01Z' }
    )
#>

  [CmdletBinding(PositionalBinding = $false)]
  param(
    [Parameter(Mandatory, ValueFromPipeline)]
    [string[]] $InputObject,
    [char] $Delimiter = ',',
    [switch] $AsSourceCode
  )
  begin {
    $allLines = ''
  }
  process {
    if (-not $allLines) {
      $allLines = $InputObject -join "`n"
    }
    else {
      $allLines += "`n" + ($InputObject -join "`n")
    }
  }
  end {

    $header, $dataLines = $allLines -split '\r?\n'

    # Parse the header line in order to derive the column (property) names.
    [string[]] $colNames = ($header, $header | ConvertFrom-Csv -ErrorAction Stop -Delimiter $Delimiter)[0].psobject.Properties.Name
    [string[]] $colTypeNames = , 'string' * $colNames.Count
    [type[]] $colTypes = , $null * $colNames.Count
    $mustReType = $false; $mustRebuildHeader = $false

    if (-not $dataLines) { throw "No data found after the header line; input must be valid CSV data." }

    foreach ($i in 0..($colNames.Count - 1)) {
      if ($colNames[$i] -match '^\[([^]]+)\]\s*(.*)$') {
        if ('' -eq $Matches[2]) { throw "Missing column name after type specifier '[$($Matches[1])]'" }
        if ($Matches[1] -notin 'string', 'System.String') {
          $mustReType = $true
          $colTypeNames[$i] = $Matches[1]
          try {
            $colTypes[$i] = [type] $Matches[1]
          }
          catch { throw }
        }
        $mustRebuildHeader = $true
        $colNames[$i] = $Matches[2]
      }
    }
    if ($mustRebuildHeader) {
      $header = $(foreach ($colName in $colNames) { if ($colName -match [regex]::Escape($Delimiter)) { '"{0}"' -f $colName.Replace('"', '""') } else { $colName } }) -join $Delimiter
    }

    if ($AsSourceCode) {
      # Note: To make the output suitable for direct piping to Invoke-Expression (which is helpful for testing),
      #       a *single* string mut be output.
    (& {
        "@("
        & { $header; $dataLines } | ConvertFrom-Csv -Delimiter $Delimiter | ForEach-Object {
          @"
    [pscustomobject] @{ $(
    $(foreach ($i in 0..($colNames.Count-1)) {
      if (($propName = $colNames[$i]) -match '\W') {
        $propName = "'{0}'" -f $propName.Replace("'", "''")
      }
      $isString = $colTypes[$i] -in $null, [string]
      $cast = if (-not $isString) { '[{0}] ' -f $colTypeNames[$i] }
      $value = $_.($colNames[$i])
      if ($colTypes[$i] -in [bool] -and ($value -as [int]) -notin 0, 1) { Write-Warning "'$value' is interpreted as `$true - use 0 or 1 to represent [bool] values."  }
      if ($isString -or $null -eq ($value -as [double])) { $value = "'{0}'" -f $(if ($null -ne $value) { $value.Replace("'", "''") }) }
      '{0} = {1}{2}' -f $colNames[$i], $cast, $value
    }) -join '; ') }
"@
        }
        ")"
      }) -join "`n"
    }
    else {
      if (-not $mustReType) {
        # No type-casting needed - just pass the data through to ConvertFrom-Csv
        & { $header; $dataLines } | ConvertFrom-Csv -ErrorAction Stop -Delimiter $Delimiter
      }
      else {
        # Construct a class with typed properties matching the CSV input dynamically
        $i = 0
        @"
class __ConvertFromCsvTypedHelper {
$(
  $(foreach ($i in 0..($colNames.Count-1)) {
    '  [{0}] ${{{1}}}' -f $colTypeNames[$i], $colNames[$i]
  }) -join "`n"
)
}
"@ | Invoke-Expression

        # Pass the data through to ConvertFrom-Csv and cast the results to the helper type.
        try {
          [__ConvertFromCsvTypedHelper[]] (& { $header; $dataLines } | ConvertFrom-Csv -ErrorAction Stop -Delimiter $Delimiter)
        }
        catch { $_ }
      }
    }
  }
}
mklement0
  • 382,024
  • 64
  • 607
  • 775
  • Thanks. I like the idea of being able to leave out the quotes altogether, but then I wasn't able to get to expected output. – not2qubit Dec 04 '22 at 00:26
  • @not2qubit, you _can_ use quotes, _if needed_. In what way weren't you able to get the expected output? Note that the answer states, "Use `"..."` around field values that themselves contain `,`" – mklement0 Dec 04 '22 at 00:27
  • No worry, it was my lack of knowing that I needed to use double `$` wrapped in `()` for in-text variables, so I didn't get any output unless I put it outside of the Write-Host string. – not2qubit Dec 04 '22 at 00:40
  • 1
    I see, @not2qubit - you can find a summary of PowerShell's string-interpolation rules in [this answer](https://stackoverflow.com/a/40445998/45375). Generally speaking, both `Write-Host` and string interpolation are poor choices for producing for-display-formatted representations (except that `Write-Host` makes _coloring_ easy). – mklement0 Dec 04 '22 at 00:43
  • Cool. What other *type literals* are supported? I only found [this](https://community.f5.com/t5/technical-articles/powershell-abc-s-t-is-for-type-literals/ta-p/284216), but that's not MS docs, nor mentioning CSV. – not2qubit Dec 04 '22 at 12:08
  • 1
    @not2qubit: See [this answer](https://stackoverflow.com/a/66996505/45375) for a systematic overview of PowerShell's type literals, which _in principle_ can refer to any .NET type. Here, the usable types are limited to those whose target type's values can be represented as numbers or string literals. I've updated the answer accordingly, along with enumerating common usable types, and instructions on how you can test whether a given type can be used. – mklement0 Dec 04 '22 at 14:39
1

Here are a few techniques that might help you use data in CSV format. I've changed your input a little. Instead of defining a separate header, I've included the header record as the first line of the CSV data. that's what ConvertFrom-CSV expects. I also changed single quotes into double quotes. And I omitted one field completely.

The first output shows what happens if you feed the output of ConvertFrom-CSV to format-List. I don't recommend that you do this if your plan is to use the data in variables. format-list is suitable for display, but not further processing.

The second output mimics your sample output. The here string contains various subexpressions, each of which can access the current data via the automatic variable $_.

Last, I show you the members of the pipeline stream. Note the four properties that got their names from your field names.

$mycsv = @"
name, id, type, loc
"Brave", "Brave.Brave", 1, "winget"
"Adobe Acrobat (64-bit)", "{AC76BA86-1033-1033-7760-BC15014EA700}", 2,
"GitHub CLI", "GitHub.cli", 3, "C:\portable"
"@

ConvertFrom-CSV $mycsv | Format-List

ConvertFrom-Csv $mycsv | % {@"
App Name: $($_.name)
App Type: $($_.type)
App id  : $($_.id)
App Loc : $($_.loc)
$("-"*40)
"@
}

ConvertFrom-CSV $mycsv | gm
Walter Mitty
  • 18,205
  • 2
  • 28
  • 58
  • Thank you Walter, pretty close to what I was looking. Really helpful to see the other approach as well, and also learning about `Get-Member`. Only question is why the powershell community seem to love double quotes in favor of single quotes. Any particular reason for this? – not2qubit Dec 04 '22 at 00:23
  • PS. When using `ConvertFrom-CSV` you get a lot of flexibility in regard to indentation and spaces and doesn't need any quotes (unless there is a comma `,`) in the original CSV text blob. – not2qubit Dec 04 '22 at 01:14
  • 1
    Powershell uses both single and double quotes, for slightly different purposes. CSV has double quotes defined, but not single quotes. This is true for ConvertFrom-CSV, but it's also true when other engines like Excel are reading CSV data. – Walter Mitty Dec 04 '22 at 12:54
  • 1
    BTW, you may be interested in a little tool that reads a CSV file and interprets the contents as a driver table for repeated expansion of a template. I have posted an early version of the tool [Here](https://stackoverflow.com/questions/42230306/how-to-combine-a-template-with-a-csv-file-in-powershell). – Walter Mitty Dec 04 '22 at 14:06
  • 1
    @not2qubit, as for loving double quotes: the issue is unrelated to PowerShell: it is the CSV standard that recognizes _only_ double quotes (but they aren't mandatory). – mklement0 Dec 04 '22 at 14:41
  • I offer the current version of Expand-Csv in Github, at [this location](https://github.com/DCressey/Expand-Csv-Powershell-function). – Walter Mitty Dec 05 '22 at 13:57