2

Originally, this question was prompted while investigating an answer to question 27373126 about selecting items from an array of hash tables. (This was about shorter ways to get the properties than using Foreach-Object or Select-Object with hash table arguments.)

Having found that I could get items using script-block arguments to Select-Object as follows:

$hash = @{title="title1";detail="detail1"},@{title="title2";detail="detail2"}
$hash | select-object {$_.title},{$_.detail}

and that I could tidy up the output objects (produced with properties named "$_.title" and "$_.detail") by defining functions as follows:

function title{$_.title};function detail{$_.detail}
$hash | select {title},{detail}

I then tried to automate producing output with properties defined by a list of property names (I know these are wrong, this is just how I found the issue I am questioning):

'title','detail' | foreach { $hash | select $_ }

title
-----





# 4 blank lines

I know why there are 4 blank lines. There is no "title" member in $hash, it is a "title" item. The surprising part was no column for "detail". Tried this:

'title','detail' | foreach { $hash | select $_ } | gm

   TypeName: Selected.System.Collections.Hashtable

Name        MemberType   Definition
----        ----------   ----------
Equals      Method       bool Equals(System.Object obj)
GetHashCode Method       int GetHashCode()
GetType     Method       type GetType()
ToString    Method       string ToString()
title       NoteProperty object title=null

Maybe something strange with Foreach-Object providing arguments to Select-Object. Tried:

$hash | select title; $hash|select detail

title
-----





# 4 blank lines

Which command is generating the output?:

'start';$hash|select title;'in';$hash|select detail;'done'
start

title
-----


in


done

So 2 blank lines from each section but no "detail" member. Time to get a bit more specific:

'start';$hash|select title|gm;'in';$hash|select detail|gm;'done'
start


   TypeName: Selected.System.Management.Automation.Internal.Host.InternalHost

Name        MemberType   Definition
----        ----------   ----------
Equals      Method       bool Equals(System.Object obj)
GetHashCode Method       int GetHashCode()
GetType     Method       type GetType()
ToString    Method       string ToString()
title       NoteProperty object title=null
in
Equals      Method       bool Equals(System.Object obj)
GetHashCode Method       int GetHashCode()
GetType     Method       type GetType()
ToString    Method       string ToString()
detail      NoteProperty object detail=null
done

So the second select IS generating a "detail" member but it is getting lost somewhere. How about this:

&{$hash|select title;$hash|select detail}|gm


   TypeName: Selected.System.Management.Automation.Internal.Host.InternalHost

Name        MemberType   Definition
----        ----------   ----------
Equals      Method       bool Equals(System.Object obj)
GetHashCode Method       int GetHashCode()
GetType     Method       type GetType()
ToString    Method       string ToString()
title       NoteProperty object title=null

No "detail" again. After lots of fruitless testing, some useful research uncovered a partial answer. The full description is in How PowerShell Formatting and Outputting REALLY works but the upshot is that the final display was being controlled by the properties of the first object through the pipeline. This object had a "title" property but no "detail" property so the output was formatted to match. That explains why the displayed output had no "detail" column but why did the last Get-Member not show it? Get-Member should have received 2 objects with a "title" followed by 2 objects with a "detail". I know this because a manual Get-Member does show them:

&{$hash|select title;$hash|select detail}|foreach{[componentmodel.typedescriptor]::getproperties($_)[0].name}
title
title
detail
detail

# Can't use gettype() since doesn't know about added properties

So it would seem that not just Format-Table decides its output based on the first object down the pipeline. Apparently, Get-Member also exhibits this behaviour, though in this case it is the first object of each type. All the objects were PSCustomObjects but the first had only a "title" added property so Get-Member assumed that all PSCustomObjects looked the same.

And finally, THE QUESTION. Which other cmdlets, if any, exhibit this "first object down the pipeline silently defines subsequent behaviour" functionality?

mklement0
  • 382,024
  • 64
  • 607
  • 775
Uber Kluger
  • 394
  • 1
  • 11
  • 2
    The answer is all of them. The first object output from a returnable command/expression determines the `format-table` table configuration. Subsequent objects output from the same command or same expression will show their values of only the properties output by the first object. – AdminOfThings Jun 26 '20 at 18:36
  • Why not convert the array of hashtables into PSObjects: `$hash | ForEach-Object { [PsCustomObject]$_ }` (if you would use [ordered] hashtables, it would show the title and values in the same order. – Theo Jun 26 '20 at 18:55
  • As an aside: `@{title="title1";detail="detail1"},@{title="title2";detail="detail2"} | Select-Object title, detail` works fine - no need for script-block / hashtable-based calculated properties. If you want to determine the properties to select dynamically, just pass an array variable to `Select-Object`: `$props = 'title', 'detail'; @{title="title1";detail="detail1"},@{title="title2";detail="detail2"} | Select-Object $props` – mklement0 Jun 26 '20 at 20:07
  • @AdminOfThings: The question is "which cmdlets behave like Format-Table when objects are passed into them?" not "which cmdlets are affected by the behaviour of Format-Table on their output?" – Uber Kluger Jun 26 '20 at 21:23
  • 1
    The answer is still potentially all of them. When you return 4 or less properties, the shell formatter outputs in table format. When 5 or more properties are returned, the formatter uses a list format. – AdminOfThings Jun 26 '20 at 21:25
  • @Theo (and AdminOfThings again): The question is about cmdlet behaviour. Processing of hash tables was the question (#27373126) that I was answering which revealed the issue in this question. – Uber Kluger Jun 26 '20 at 21:28
  • 1
    The answer revealed in [27373126](https://stackoverflow.com/questions/27373126/select-item-from-powershell-hashtable-without-using-foreach) is using a [calculated property](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/select-object?view=powershell-7). Calculated properties are used in conjunction with `Select-Object`. It is a hash table but it has specific keys that must always be present to which you can supply values. You can't make up your own keys like `Title` and `Detail`. – AdminOfThings Jun 26 '20 at 21:32
  • @mklement0: You might like to clarify which version you are using. Entered your test into V5.1.18362.752 and received 2 correctly titled blank lines. – Uber Kluger Jun 26 '20 at 21:34
  • 1
    @AdminOfThings: One last time (no flame wars). This is not about answering question 27373126, it is about something essentially unrelated that I discovered WHILE answering that question. I gave mklement0 an upvote because actual cmdlets were listed. I did not mark it as answered because the list is probably incomplete. I am not sure if a complete list that doesn't become out of date as new cmdlets are added is possible. Perhaps my question is essentially rhetorical and should be taken as a warning that some cmdlets do this so be careful piping objects of the same type but with varying members. – Uber Kluger Jun 26 '20 at 21:47
  • 1
    It is not possible to maintain or even make a static list. You just have to learn by experience. We can only request a feature change if we don't like something or question specific anomalies to see if they should be changed. You should just assume all cmdlets behave this way. When you run into one that doesn't, then rejoice and remember it. – AdminOfThings Jun 26 '20 at 21:53
  • 1
    Good point, @UberKluger: My commands work in PowerShell [Core] v6+; in Windows PowerShell you indeed need `[pscustomobject]` instance as `Select-Object` input: `[pscustomobject] @{title="title1";detail="detail1"}, [pscustomobject] @{title="title2";detail="detail2"} | Select-Object title, detail`. Passing the property names via an array variable to `Select-Object` works in both editions, however. – mklement0 Jun 26 '20 at 22:05
  • 1
    @UberKluger, re "because the list is probably incomplete. I am not sure if a complete list that doesn't become out of date as new cmdlets are added is possible" - my answer gives you the _criterion_ for identifying the cmdlets that exhibit the first-input-object-determines-the-behavior-for-all-input-objects behavior, along with notable examples. It is in the nature of your question that this list may grow in the future (and it may apply to various _third-party_ cmdlets). This is not a shortcoming of the answer, so I suggest you accept it. – mklement0 Jun 26 '20 at 22:15
  • 1
    @UberKluger I understand what you want to know with your question. You want to know which commands, when piped into or provided input, lose track of properties of the input objects when the properties list is not homogeneous. The question is good and valid. The examples provided in the post though make it difficult to decipher what you want because your code incorrectly exhibits the behavior you want explained. Hash tables and custom objects are not the same. Using custom objects instead would have made this a lot clearer, at least for me. – AdminOfThings Jun 26 '20 at 22:21
  • 1
    @AdminOfThings Agreed, and there's a lot of incidental information in the question, but note that as of at least v7.0 piping hash tables to `Select-Object` works fine, and the predictable, ordered property selection provided by `Select-Object` makes it irrelevant that the entries of the input hash table are inherently unordered. – mklement0 Jun 26 '20 at 22:28
  • 2
    @mklement0, good point. I think the posted answer covers everything in the comments anyway. All of these comments should probably just be purged. They don't add much that isn't covered in the answer. – AdminOfThings Jun 26 '20 at 22:31
  • @UberKluger To finish the `Select-Object` story: With a preexisting array of hash tables in v6-, use `.ForEach()` as follows (casting the whole array to `[pscustomobject[]]` does _not_ work, not even in v7): `$hts = @{title="title1";detail="detail1"}, @{title="title2";detail="detail2"}; $hts.ForEach({ [pscustomobject] $_ }) | Select-Object title, detail`. Of course, you can also construct the custom objects directly inside the script block, without `Select-Object`: `$hts.ForEach({ [pscustomobject] @{ title = $_.title; detail = $_.detail } })` – mklement0 Jun 27 '20 at 14:43
  • @AdminOfThings custom objects might have made it clearer but the original question 27373126 was about hash tables so that's what I was using in my testing. Also, it was the output from Select-Object (as rendered by Format-Table) that was confusing and this is an Object (Selected.System.Collections.Hashtable) essentially the same as (if not identical to) a custom object with the same properties. Again, my question was about cmdlets that unexpectedly and silently discard properties (pretty much answered). Suggestions about hash table selection are better attached to question 27373126 . – Uber Kluger Jun 28 '20 at 20:00
  • @AdminOfThings "It is not possible to maintain...", true except that it is possible (sort of). While a list of cmdlets is impractical, how about if the cmdlets (or at least their documentation) did it themselves. Take Export-Csv (see comment below answer), would it not be possible to clearly describe what it does, e.g. "The list of headings is derived from the properties of the first object received and only these properties will be listed for any subsequent objects received." At least then, when you get unexpected results you can easily find out why just by a careful reading of the manual. – Uber Kluger Jun 28 '20 at 20:11

1 Answers1

4

Note: In PowerShell versions before 7.0, hash tables indeed aren't supported as input to Select-Object; in earlier versions, cast them to [pscustomobject] first; e.g.:
[pscustomobject] @{title="title1";detail="detail1"}, [pscustomobject] @{title="title2";detail="detail2"} | Select-Object title, detail

To recap the problem that you're already aware of: (possibly implicit) Format-Table output locks in the properties of the first object as display columns and subsequent objects that do not have these properties will show blank column values - see this answer for details.
It is important to note that this is merely a display problem.

The Get-Member problem is slightly different: it isn't about the first input object's type, it's about input objects that share the same type: Get-Member is designed to produce output for each distinct type among the input objects.

The problem is that two [pscustomobject] instances (such as output by Select-Object) are considered the same type by Get-Member (which, from a .NET perspective, they technically are) - even if they have different properties, so Get-Member will in effect only show the members of the first [pscustomobject] instance among the input objects (which may or may not be the first input object overall).

Therefore, in order to pipe multiple [pscustomobject] instances to Get-Member and see the properties of each, call Get-Member via ForEach-Object:

# Two sample [pscustomobject] instances with non-overlapping properties.
$objs = [pscustomobject] @{ foo = 1 }, [pscustomobject] @{ bar = 2 }
# Call Get-Member on *each* object.
$objs | ForEach-Object { Get-Member -InputObject $_ }

As for:

THE QUESTION. Which other cmdlets, if any, exhibit this "first object down the pipeline silently defines subsequent behaviour" functionality?

In addition to (potentially implicitly applied) Format-Table, in essence all cmdlets that expect all input objects to be of the same type (have the same set of properties) exhibit the behavior, notably Export-Csv / ConvertTo-Csv. Any number of third-party cmdlets may fall into that category as well - there's no way to come up with an exhaustive list, not least because new cmdlets may be introduced in the future.

You can generally infer from the purpose of a cmdlet whether it requires uniform input or not (and, hopefully, the documentation makes it clear too):

  • Export-Csv and ConvertTo-Csv, because they create tabular data, require uniform input; while you can use -Force with -Append to make Export-Csv append objects that also have different properties to a preexisting CSV file, only those properties present on the newly added objects that match the preexisting columns (properties) are added, so uniformity is ultimately still enforced.

  • By contrast, ConvertTo-Json, because it can serialize arbitrary object graphs, does not require uniform input.

mklement0
  • 382,024
  • 64
  • 607
  • 775
  • 1
    "all cmdlets ... exhibit the behaviour" defines which cmdlets match the question but knowing if a cmdlet **expects** objects to be the same is hard. Buried way down in the Export-CSV manual "When you submit multiple objects to Export-CSV, Export-CSV organizes the file based on the properties of the first object that you submit. If the remaining objects do not have one of the specified properties, the property value of that object is null, as represented by two consecutive commas. If the remaining objects have additional properties, those property values are not included in the file." – Uber Kluger Jun 28 '20 at 22:19
  • 1
    And then there's "When Force and Append parameters are combined, objects that contain mismatched properties can be written to a CSV file. Only the properties that match are written to the file. The mismatched properties are discarded." which might be taken as implying that it looks at all the objects to determine the listed properties but in fact means that **none** of the objects are examined and the properties are selected to match the existing properties in the file. Anyway, I suppose taking all that's been said in this question, this is probably the closest to an "answer" that we can get. – Uber Kluger Jun 28 '20 at 22:24