How to serialize an object in powershell to json and get identical result in PS desktop and core?

Question

Prolog

It turns out that in my case it is important to understand the source of the objects - it is a JSON payload from a REST API response. Unfortunately, JSON -> Object conversion produces different results on PS desktop vs PS core. On desktop the numbers are deserialized into Int32 types, but on core - to Int64 types. From that it follows that I cannot use Export-CliXml, because the binary layout of the objects is different.

Main question

I have a unit test that needs to compare the actual result with an expected. The expected result is saved in a json file, so the procedure is:

1. Convert the actual result to json string
2. Read the expected result from disk to string
3. Compare the actual and the expected as strings

Unfortunately, this scheme does not work because PS desktop ConvertTo-Json and PS core ConvertTo-Json do not produce identical results. So, if the expected result was saved on desktop and the test runs on core - boom, failure. And vice versa.

One way is to keep two versions of jsons. Another way is to use a library to create the json.

First I tried the Newtonsoft-Json powershell module, but it just does not work. I think the problem is that whatever C# library we use, it must be aware of PSCustomObject and alike and treat them specially. So, we cannot just take any C# JSON library.

At this point I am left with having two jsons - one per PS edition, which is kind of sad.

Are there better options?

EDIT 1

I guess I can always read the json, convert to object and then back to json again. That sucks.

EDIT 2

I tried to use ConvertTo-Json -Compress. This eliminates the difference in spacing, but the problem is that for some reason the desktop version translates all the non characters to \u000... representation. The core version does not do it.

Please, observe:

Desktop

C:\> @{ x = "'a'" } |ConvertTo-Json -Compress
{"x":"\u0027a\u0027"}
C:\>

Core

C:\>  @{ x = "'a'" } |ConvertTo-Json -Compress
{"x":"'a'"}
C:\>

Now the core version has the flag -EscapeHandling, so:

C:\>  @{ x = "'a'" } |ConvertTo-Json -Compress -EscapeHandling EscapeHtml
{"x":"\u0027a\u0027"}
C:\>

Bingo! Same result. But now this code does not run on the desktop version, which does not have this flag. More massaging is needed. I will check if that is the only problem.

EDIT 3

It is impossible to reconcile the differences between the core and the desktop versions without expensive post processing. Please, observe:

Desktop

C:\> @{ x = '"a"';y = "'b'" } |ConvertTo-Json -Compress
{"y":"\u0027b\u0027","x":"\"a\""}
C:\>

Core

C:\> @{ x = '"a"';y = "'b'" } |ConvertTo-Json -Compress -EscapeHandling EscapeHtml
{"y":"\u0027b\u0027","x":"\u0022a\u0022"}
C:\> @{ x = '"a"';y = "'b'" } |ConvertTo-Json -Compress
{"y":"'b'","x":"\"a\""}
C:\>

Any suggestions on how to salvage the json approach?

EDIT 4

The Export-CliXml approach does not work too, because of the differences between the PS versions.

Desktop

C:\> ('{a:1}' | ConvertFrom-Json).a.gettype()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Int32                                    System.ValueType


C:\>

Core

C:\> ('{a:1}' | ConvertFrom-Json).a.gettype()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Int64                                    System.ValueType

C:\>

So the same JSON is represented using different numeric types - Int32 in desktop and Int64 in core. That puts to bed the option of using Export-CliXml.

Unless I am missing something.

I believe there is no other choice, but do the double conversion - json -> object -> json and then I will have two jsons created on the same PS edition. That sucks big time.

Answer 1

• On converting from the original JSON, use the third-party Newtonsoft.Json PowerShell wrapper's ConvertFrom-JsonNewtonsoft cmdlet - this should ensure cross-edition compatibility (which the built-in ConvertFrom-Json does not guarantee across PowerShell editions, because Windows PowerShell uses a custom parser, whereas PowerShell [Core] v6+ uses Newtonsoft.json up to at least v7.1, though a move to the new(ish) .NET Core System.Text.Json API is coming).

  • Important: ConvertFrom-JsonNewtonsoft returns (arrays of) nested ordered hashtables ([ordered] @{ ... }, System.Collections.Specialized.OrderedDictionary), unlike the nested [pscustomobject] graphs that the built-in ConvertFrom-Json outputs. Similarly, ConvertTo-JsonNewtonsoft expects only (arrays of) hashtables (dictionaries) as input, and notably does not support [pscustomobject] instances, as you've learned yourself.

    • Caveat: As of this writing, the wrapper module was last updated in May 2019, and the version of the underlying bundled Newtonsoft.Json.dll assembly is quite old (8.0, whereas 12.0 is current as of this writing). See the module's source code.

  • Note that in order to parse JSON obtained from a RESTful web service manually, you mustn't use Invoke-RestMethod, as it implicitly parses and returns [pscustomobject] object graphs. Instead, use Invoke-WebRequest and access the returned response's .Content property.

• On converting to a format suitable for storing on disk, you have two options:

  • (A) If you do need the serialized format to be JSON also, you must convert all [pscustomobject] graphs to (ordered) hashtables before passing them to ConvertTo-JsonNewtonsoft.

    • See below for function ConvertTo-OrderedHashTable, which does just that.

  • (B) If the specific serialization format isn't important, i.e. if all that matters is that the formats are identical across PowerShell editions for the purpose of comparison, no extra works is needed: use the built-in Export-Clixml cmdlet, which can handle any type and produces PowerShell's native, XML-based serialization format called CLIXML (as notably used in PowerShell remoting), which should be cross-edition-compatible (at least with v5.1 on the Windows PowerShell side and as of PowerShell [Core] v7.1, both of which use the same version of the serialization protocol, 1.1.0.1, as reported by $PSVersionTable.SerializationVersion).

    • While you could re-convert such a persisted file to objects with Import-Clixml, the potential loss of type fidelity on deserialization makes comparing the serialized (CLIXML) representations advisable.

    • Also note that as of PowerShell v7.1 there is no cmdlet-based way to create an in-memory CLIXML representation, so you'll have to use the PowerShell API directly for now: System.Management.Automation.PSSerializer.Serialize. However, providing in-memory counterparts to Import-CliXml / Export-CliXml in the form of ConvertFrom-CliXml / ConvertTo-CliXml cmdlets has been green-lighted as a future enhancement.

---

Re (A): Here's function ConvertTo-OrderedHashtable, which converts (potentially nested) [pscustomobject] objects to ordered hashtables while passing other types through, so you should be able to simply insert it into a pipeline as follows:

# CAVEAT: ConvertTo-JsonNewtonSoft only accepts a *single* input object.
[pscustomobject] @{ foo = 1 }, [pscustomobject] @{ foo = 2 } | 
  ConvertTo-OrderedHashtable |
    ForEach-Object { ConvertTo-JsonNewtonSoft $_ }

function ConvertTo-OrderedHashtable {
<#
.SYNOPSIS
Converts custom objects to ordered hashtables.

.DESCRIPTION
Converts PowerShell custom objects (instances of [pscustomobject]) to
ordered hashtables (instances of [System.Collections.Specialized.OrderedDictionary]),
which is useful for to-JSON serialization via the Newtonsoft.JSON library.

Note: 
 * Custom objects are processed recursively.
 * Any scalar non-custom objects are passed through as-is.
 * Any (non-dictionary) collections in property values are converted to 
  [object[]] arrays.

.EXAMPLE
1, [pscustomobject] @{ foo = [pscustomobject] @{ bar = 'none' }; other = 2 } | ConvertTo-OrderedHashtable

Passes integer 1 through, and converts the custom object to a nested ordered
hashtable.
#>
  [CmdletBinding()]
  param(
    [Parameter(ValueFromPipeline)] $InputObject
  )

  begin {

    # Recursive helper function
    function convert($obj) {
      if ($obj -is [System.Management.Automation.PSCustomObject]) {
        # a custom object: recurse on its properties
        $oht = [ordered] @{ }
        foreach ($prop in $obj.psobject.Properties) {
          $oht.Add($prop.Name, (convert $prop.Value))
        }
        return $oht
      }
      elseif ($obj -isnot [string] -and $obj -is [System.Collections.IEnumerable] -and $obj -isnot [System.Collections.IDictionary]) {
        # A collection of sorts (other than a string or dictionary (hash table)), recurse on its elements.
        return @(foreach ($el in $obj) { convert $el })
      }
      else { 
        # a non-custom object, including .NET primitives and strings: use as-is.
        return $obj
      }
    }

  }

  process {

    convert $InputObject

  }

}

---

Re (B): A demonstration of the Export-CliXml approach (you can run this code from either PS edition):

$sb = {

  Install-Module -Scope CurrentUser Newtonsoft.json
  if (-not $IsCoreClr) { 
    # Workaround for PS Core's $env:PSModulePath overriding WinPS'
    Import-Module $HOME\Documents\WindowsPowerShell\Modules\newtonsoft.json
  }
  @'
  {
      "results": {
          "users": [
              {
                  "userId": 1,
                  "emailAddress": "jane.doe@example.com",
                  "date": "2020-10-05T08:08:43.743741-04:00",
                  "attributes": {
                      "height": 165,
                      "weight": 60
                  }
              },
              {
                  "userId": 2,
                  "emailAddress": "john.doe@example.com",
                  "date": "2020-10-06T08:08:43.743741-04:00",
                  "attributes": {
                      "height": 180,
                      "weight": 72
                  }
              }
          ]
      }
  }
'@ | ConvertFrom-JsonNewtonsoft | Export-CliXml "temp-$($PSVersionTable.PSEdition).xml"
  
}
  
# Execute the script block in both editions

Write-Verbose -vb 'Running in Windows PowerShell...'
powershell -noprofile $sb

Write-Verbose -vb 'Running in PowerShell Core...'
pwsh -noprofile $sb
  
# Compare the resulting CLIXML files.
Write-Verbose -vb "Comparing the resulting files: This should produce NO output,`n         indicating that the files have identical content."
Compare-Object (Get-Content 'temp-Core.xml') (Get-Content 'temp-Desktop.xml')

Write-Verbose -vb 'Cleaning up...'
Remove-Item 'temp-Core.xml', 'temp-Desktop.xml'

You should see the following verbose output:

VERBOSE: Running in Windows PowerShell...
VERBOSE: Running in PowerShell Core...
VERBOSE: Comparing the resulting files: This should produce NO output, 
         indicating that the files have identical content.
VERBOSE: Cleaning up...