1

I have this function that gets the content of a json file. I'm having some (to me) unexpected behavior trying to pipe this to ConvertTo-Json and Foreach-Object

Function GetConfigJson
{
    $ConfigPath = "pathtomyjsonfile.json"
    return Get-Content $ConfigPath | Out-String
}

The json is formatted like [{"key1":"value1"}, {"key2":"value2"}, ...]

To test the behavior I did the following:

$a = 0;
GetConfigJson | ConvertFrom-Json | ForEach-Object { $a++ };
$b = 0;
ConvertFrom-Json GetConfigJson | ForEach-Object { $b++ };
$c = 0;
ConvertFrom-Json (GetConfigJson) | ForEach-Object { $c++ };
$d = 0;
(ConvertFrom-Json (GetConfigJson)) | ForEach-Object { $d++ };

Write-Host "Test1: $a | Test2: $b | Test3: $c | Test4: $d";

Out of these only Test4 prints the expected number, Test1 and Test3 print 1 and Test2 gets an error: ConvertFrom-Json : Invalid JSON primitive: GetConfigJson.

Why do I need the parentheses around the ConvertFrom-Json for it to actually get piped as an array of objects?

(The parentheses around the function name GetConfigJson is more acceptable - but I still wonder why I need it there?)

Ghork
  • 70
  • 8
  • I think that the actual "bracket issue" is within your function; If you want to [return an expression](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_return?view=powershell-7#syntax), you will need to put that in brackets: `return (Get-Content $ConfigPath | Out-String)`. But without an input example it is hard to confirm your results. Anyways, the `Return` is not required because ***In PowerShell, the results of each statement are returned as output, even without a statement that contains the Return keyword***. – iRon Aug 13 '20 at 10:19
  • And the `| Out-String` is also not required as [`ConvertFrom-Json`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertfrom-json?view=powershell-7#parameters) (`-InputObject`) accepts a streamed list of lines (`String[]`), it will only result in stalling the pipeline. Meaning you might just leave the function and simplify this to: `Get-Content $ConfigPath | ConvertFrom-Json | ForEach-Object { $a++ }` – iRon Aug 13 '20 at 10:28

1 Answers1

3

It might help to take a look at the types of the output from each example - see below for a breakdown.

Helper Functions

I'm using these the two helper functions in the sections below. Note - I'm guessing your config has an array at the root as that seems to reproduce the issue, but feel free to update your question if that's not true.

function GetConfigJson
{
    return "[{""name"":""first""}, {""name"":""second""}]"
}

function Write-Value
{
    param( $Value )
    write-host $Value.GetType().FullName
    write-host (ConvertTo-Json $Value -Compress)
}

And then using your examples:

Example 1

# example 1a - original example
PS> $a = 0
PS> GetConfigJson | ConvertFrom-Json | ForEach-Object { $a++ };
PS> $a
1

# example 1b - show return types and values
PS> GetConfigJson | ConvertFrom-Json | foreach-object { Write-Value $_ }
System.Object[]
{"value":[{"name":"first"},{"name":"second"}],"Count":2}

ConvertFrom-Json returns an array object with two entries, but Foreach-Object only runs once because it iterates over the single array object, not the 2 items in the array.

Example 2

# example 2a - original example
PS> $b = 0;
PS> ConvertFrom-Json GetConfigJson | foreach-object { $b++ }
ConvertFrom-Json : Invalid JSON primitive: GetConfigJson.
At line:1 char:1
+ ConvertFrom-Json GetConfigJson | foreach-object { $b++ }
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [ConvertFrom-Json], ArgumentException
    + FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

# example 2b - show parameter types and values
PS> Write-Value GetConfigJson
System.String
"GetConfigJson"

ConvertFrom-Json throws an exception because PowerShell is treating GetConfigJson as a literal string, but "GetConfigJson" obviously isn't valid json, hence the exception.

Example 3

# example 3a - original example
PS> $c = 0;
PS> ConvertFrom-Json (GetConfigJson) | ForEach-Object { $c++ };
PS> $c
1

# example 3b - show parameter types and values
PS> ConvertFrom-Json (GetConfigJson) | ForEach-Object { Write-Value $_ };
System.Object[]
{"value":[{"name":"first"},{"name":"second"}],"Count":2}

This uses the Grouping Operator ( ... ) around GetConfigJson, so PowerShell evaluates GetConfigJson as a call to a function instead of taking it as a literal string. It first executes the GetConfigJson expression and then passes the result of that as a parameter into ConvertFrom-Json. However, it's still iterating over the single array object rather than over the items, so the foreach-object only runs once.

Example 4

# example 4a - original example
PS> $d = 0;
PS> (ConvertFrom-Json (GetConfigJson)) | ForEach-Object { $d++ };
PS> $d
2

# example 4b - show parameter types and values
PS> (ConvertFrom-Json (GetConfigJson)) | ForEach-Object { Write-Value $_ };
System.Management.Automation.PSCustomObject
{"name":"first"}
System.Management.Automation.PSCustomObject
{"name":"second"}

We're using the grouping operator twice here - once around GetConfigJson to evaluate that as an expression as opposed to a string, and once around the whole ConvertFrom-Json (GetConfigJson). The outer ( ... ) causes PowerShell to "unroll" the single array object and emits its items into the pipeline consumed by Foreach-object. This means ForEach-Object iterates over the items and we see two separate values written out by ``Write-Value```

Summary

You managed to hit a lot of PowerShell quirks with this question - hopefully this answer helps understand what they're all doing and why you see the behaviour you do.

Update for PowerShell 7

Per the comment below from @mklement0, the behaviour of ConvertFrom-Json changes from version 7 onwards - it doesn't enumerate arrays by default, and requires -NoEnumerate to opt out.

E.g., '[ 1, 2 ]' | ConvertFrom-Json | Measure-Object now reports 2 in v7+, whereas -NoEnumerate is required to get the v6- behaviour: '[ 1, 2 ]' | ConvertFrom-Json -NoEnumerate | Measure-Object (reports 1).

mclayton
  • 8,025
  • 2
  • 21
  • 26
  • So to avoid `Example 2` could I also do something like `$function:GetConfigJson` ? Is there another way to "unroll" the single array? What I don't understand is why it is a single array to begin with. If you store it in a variable would it also unroll? – Ghork Aug 13 '20 at 12:26
  • ```$function:GetConfigJson``` returns a *scriptblock* that represents the executable code in the body of the function, but it doesn't actually execute the function, so you'd still get an error with ```ConvertFrom-Json $function:GetConfigJson | foreach-object { $b++ }```- it'd just be a different error :-) – mclayton Aug 13 '20 at 12:44
  • Re the array, it might be worth updating your question with an example config that demonstrates the issue. I'm assuming your config is in the form ```[{"key":"value"},{"key":"value"}]``` so that ```ConvertFrom-Json``` returns an array object - my whole answer hinges on that being the case. If that's not true then I'd need to see an example config to know whether my answer is still valid for your situation. – mclayton Aug 13 '20 at 12:51
  • There are more keys/values within each bracket but for the purpose of this issue your format is correct. But would you say the issue is then that `ConvertFrom-Json` simply returns an array like this with length one containing another array? If so is there an explicit way I can make sure I "unroll" it? Or could I format my Json anyway to avoid this issue? – Ghork Aug 13 '20 at 13:22
  • Nicely done - I suggest mentioning that **the (non-)enumeration behavior changed in PowerShell [Core] 7.0**: `ConvertFrom-Json` now _does_ enumerate arrays by default, and requires `-NoEnumerate` to opt out. E.g., `'[ 1, 2 ]' | ConvertFrom-Json | Measure-Object` now reports `2` in v7+, whereas `-NoEnumerate` is required to get the v6- behavior: `'[ 1, 2 ]' | ConvertFrom-Json -NoEnumerate | Measure-Object` (reports `1`). – mklement0 Aug 13 '20 at 13:42
  • @ghork - ```ConvertFrom-Json``` returns exactly whatever your json describes. In your case, the json defines an array with two child items in it. The difference is how you subsequently pipe that array into ```foreach-object```. In some cases you're piping in the *array* object so the pipeline contains exactly one object - the array itself, and in other cases the array gets unrolled and you're piping the *items* from inside the array into ```foreach-object``` so the pipeline contains multiple objects. – mclayton Aug 13 '20 at 13:55
  • 1
    @mklement0 - will do. I'm still mostly stuck on PowerShell 5.1 so I might just paraphrase your comment if that's ok :-). – mclayton Aug 13 '20 at 14:05