2

I've spent the last hour or so trying to figure out what's happening here, and I simply can't figure out the cause, nor have I accidentally stumbled upon a workaround.

When I run winget upgrade the data returned is correctly encoded:

winget response correctly encoded

Yet if I store the response of winget upgrade in a variable ($winget = winget upgrade; $winget;), some characters break, and it seems like an encoding issue:

winget with encoding issue in response

cmdlets like this will always return strings to PowerShell, so I guess assigning the response to the variable is applying some default text encoding, perhaps from my PowerShell profile? I've tried casting most of the common codes on the variable, but it ends up breaking the data entirely.

I've even dumped the data out in every possible encoding type and they're all broken:

winget upgrade | Out-File -FilePath C:\tmp\test1.txt -Encoding ascii
winget upgrade | Out-File -FilePath C:\tmp\test3.txt -Encoding default
winget upgrade | Out-File -FilePath C:\tmp\test4.txt -Encoding string
winget upgrade | Out-File -FilePath C:\tmp\test5.txt -Encoding unicode
winget upgrade | Out-File -FilePath C:\tmp\test7.txt -Encoding utf32
winget upgrade | Out-File -FilePath C:\tmp\test8.txt -Encoding utf7
winget upgrade | Out-File -FilePath C:\tmp\test9.txt -Encoding utf8

I've managed to affect the faulty characters by re-encoding the string, bt hasn't fix the issue:

$originalBytes = [Text.Encoding]::Default.GetBytes('Microsoft Visual C++ 2013 Redistributable (x6')
[Text.Encoding]::Utf8.GetString($originalBytes)

enter image description here

Arbiter
  • 450
  • 5
  • 26
  • Use Set-Content instead of Out-File? – Doug Maurer Nov 17 '21 at 23:48
  • @DougMaurer same outcome, tbh I don't want to write to a file at all, just a variable. May I ask if there was a reason you suspected that might work? I'd love to know if you have the same issue with your machine? I only have the 1 device to test on – Arbiter Nov 18 '21 at 00:01
  • Oh god... unfortunately not an option for my use case, so back to the drawing board. I was really hoping I could find a way to fix that data :( thanks for chipping in! – Arbiter Nov 18 '21 at 00:07
  • 2
    A [mojibake](https://en.wikipedia.org/wiki/Mojibake) case. In Python, `[{x:'…'.encode( 'utf-8').decode( x)} for x in ['cp1252', 'cp850', 'cp437']]` returns `[{'cp1252': '…'}, {'cp850': 'ÔǪ'}, {'cp437': 'ΓǪ'}]`. Check `[console]::OutputEncoding` (note that `winget upgrade` shows correct output for `utf-8` in my box). – JosefZ Nov 18 '21 at 15:31
  • Thanks JosefZ, that is very helpful – Doug Maurer Nov 18 '21 at 18:28

2 Answers2

5

Credit to @JosefZ for sharing the solution, which can be found here: How to configure the encoding for Powershell console?

This is the output of [console]::OutputEncoding:

PS C:\> [console]::OutputEncoding                             
                                                              
                                                              
IsSingleByte      : True                                      
BodyName          : ibm850                                    
EncodingName      : Western European (DOS)                    
HeaderName        : ibm850                                    
WebName           : ibm850                                    
WindowsCodePage   : 1252                                      
IsBrowserDisplay  : False                                     
IsBrowserSave     : False                                     
IsMailNewsDisplay : False                                     
IsMailNewsSave    : False                                     
EncoderFallback   : System.Text.InternalEncoderBestFitFallback
DecoderFallback   : System.Text.InternalDecoderBestFitFallback
IsReadOnly        : True                                      
CodePage          : 850

This is the output of [Console]::Out:

PS C:\> [Console]::Out                                 
                                                       
Encoding                         FormatProvider NewLine
--------                         -------------- -------
System.Text.SBCSCodePageEncoding en-GB          ...

Changing my console output to UTF-8 using [Console]::OutputEncoding = [System.Text.Encoding]::UTF8 has solved the issue.

Now, when I store the cmdlet response in a variable, all is right in the world:

PS C:\> [Console]::OutputEncoding = [System.Text.Encoding]::UTF8 
PS C:\> $winget = winget upgrade                                 
PS C:\> $winget 
Name                                           Id                                    Version       Available     Source
-----------------------------------------------------------------------------------------------------------------------
Signal                                         OpenWhisperSystems.Signal             5.20.0        5.23.1        winget
Cisco Webex Meetings                           Cisco.CiscoWebexMeetings              41.9.5        41.12.2.15    winget
Audacity                                       Audacity.Audacity                     3.0.4         3.1.2         winget
Docker Desktop                                 Docker.DockerDesktop                  4.1.1         4.2.0         winget
Git                                            Git.Git                               2.33.0.2      2.34.0        winget
MongoDB Compass                                MongoDB.Compass.Full                  1.28.4        1.29.4.0      winget
TeamViewer                                     TeamViewer.TeamViewer                 15.22.3       15.23.9       winget
Visual Studio Community 2019                   Microsoft.VisualStudio.2019.Community 16.11.3       16.11.7       winget
Insomnia                                       Insomnia.Insomnia                     2021.5.3      2021.6.0      winget
Microsoft Visual C++ 2013 Redistributable (x6… Microsoft.VC++2013Redist-x64          12.0.30501.0  12.0.40664.0  winget
Microsoft Visual C++ 2015-2019 Redistributabl… Microsoft.VC++2015-2019Redist-x64     14.29.30133.0 14.29.30135.0 winget
Microsoft Visual C++ 2015-2019 Redistributabl… Microsoft.VC++2015-2019Redist-x86     14.29.30133.0 14.29.30135.0 winget
Node.js LTS                                    OpenJS.NodeJS.LTS                     14.17.6       16.13.0       winget
Microsoft Visual C++ 2015-2019 Redistributabl… Microsoft.VC++2015-2019Redist-x64     14.28.29914.0 14.29.30135.0 winget
ScreenToGif                                    NickeManarin.ScreenToGif              2.33.1        2.34.1        winget
MongoDB                                        MongoDB.Server                        5.0.2         5.1.0         winget
AWS Command Line Interface                     Amazon.AWSCLI                         2.2.46.0      2.3.7.0       winget
Chrome Remote Desktop Host                     Google.ChromeRemoteDesktop            92.0.4515.41  96.0.4664.39  winget
Microsoft Visual C++ 2013 Redistributable (x8… Microsoft.VC++2013Redist-x86          12.0.30501.0  12.0.40664.0  winget
19 upgrades available.
Arbiter
  • 450
  • 5
  • 26
1

I know you said replacing the characters is not an option. I'm curious to see what you come up with. For the rest of us, here is a little code that will turn the output of winget upgrade into usable object.

(winget upgrade) -replace 'ΓǪ|…|…',' ' | ForEach-Object {
    if($_ -match '(.+?)\s{2,}(.+?)\s+(\S+)\s+(\S+)\s+(\S+)$'){
        $Matches[1..5] -join '|'
    }
} | ConvertFrom-Csv -Delimiter '|'

Here is the default output in list form

Name      : Cisco Webex Meetings
Id        : Cisco.CiscoWebexMeetings
Version   : 41.9.5
Available : 41.12.2.15
Source    : winget

Name      : Discord
Id        : Discord.Discord
Version   : 0.0.309
Available : 1.0.9003
Source    : winget

Name      : GitHub Desktop
Id        : GitHub.GitHubDesktop
Version   : 2.9.0
Available : 2.9.4
Source    : winget

And when using Format-Table

Name                                       Id                                    Version         Available      Source
----                                       --                                    -------         ---------      ------
Visual Studio Community 2019               Microsoft.VisualStudio.2019.Community 16.10.31410.357 16.11.7        winget
Cisco Webex Meetings                       Cisco.CiscoWebexMeetings              41.9.5          41.12.2.15     winget
Discord                                    Discord.Discord                       0.0.309         1.0.9003       winget

I've tested in 5.1, 7.2, ISE, and Visual Studio Code.

Doug Maurer
  • 8,090
  • 3
  • 12
  • 13
  • 1
    I had already written something to do this, but my 'objectifier' code was a lot longer than yours, I took an entirely different approach. Thanks for sharing! +1 – Arbiter Nov 18 '21 at 16:35