4

I have this code in my powershell script and it doesn't do well on the special characters parts. 

 $request = 'http://151.80.109.18:8082/vrageremote/v1/session/players'
 $a = Invoke-WebRequest -ContentType "application/json; charset=utf-8" $request |
 ConvertFrom-Json    |
 Select -expand Data |
 Select -expand players |
 Select displayName, factionTag | Out-file "$scriptPath\getFactionTag.txt"

In my output file I only get '????' for any special characters. Does anyone know how I can get it to show special characters in my output file?

mklement0
  • 382,024
  • 64
  • 607
  • 775
user1425462
  • 57
  • 2
  • 2
  • 6

3 Answers3

15

Peter Schneider's helpful answer and Nas' helpful answer both address one problem with your approach: You need to:

  • either: access the .Content property on the response object returned by Invoke-WebRequest to get the actual data returned (as a JSON string), which you can then pass to ConvertFrom-Json.

  • or: use Invoke-RestMethod instead, which returns the data directly and parses it into custom objects, so you can work with these objects directly, without the need for ConvertTo-Json; however, with a character-encoding problem such as in this case this is not an option, because explicit re-encoding of the JSON string is needed - see below.

However, you still have a character-encoding problem, because, in the absence of charset information in the response header, PowerShell interprets the UTF-8-encoded JSON string returned as ISO-8859-1-encoded, in Windows PowerShell as well as in PowerShell (Core) up to v7.3.3, except that v7.0+ defaults to UTF-8 for JSON, specifically.
v7.4+ will use UTF-8 as the general default, i.e. for all media types.

There are two possible solutions:

  • Preferably, amend the web service to include charset=utf-8 in the response header's ContenType field.

  • If you can't do that, you must perform your own decoding, based on the raw bytes of the response body, accessible via the .RawContentStream property:

Here's the implementation of the latter:

# Note that there's no point in using 
# -ContentType  "application/json; charset=utf-8" in this case,
# as -ContentType only applies to data sent *to* the web service.
$request = 'http://151.80.109.18:8082/vrageremote/v1/session/players'
$a = Invoke-WebRequest $request

# $a.Content cannot be used, because it contains the *misinterpreted* JSON string,
# but $a.RawContentStream provides access to the raw bytes,
# which you can decode into a string with the encoding of choice.
$jsonCorrected = 
  [Text.Encoding]::UTF8.GetString(
    $a.RawContentStream.ToArray()
  )

# Now process the reinterpreted string.
$jsonCorrected |
  ConvertFrom-Json    |
  Select -expand Data |
  Select -expand players |
  Select displayName, factionTag | Out-file "$scriptPath\getFactionTag.txt"

Note:

  • This answer provides convenience function ConvertTo-BodyWithEncoding, which wraps the functionality above.
mklement0
  • 382,024
  • 64
  • 607
  • 775
4

Use Invoke-RestMethod if you only need the json data without the ParsedHtml, Headers and other objects returned by Invoke-WebRequest

$request = 'http://151.80.109.18:8082/vrageremote/v1/session/players'
$a = Invoke-RestMethod -ContentType "application/json; charset=utf-8" $request |
Select -expand Data |
Select -expand players |
Select displayName, factionTag | Out-file "$scriptPath\getFactionTag.txt"
Nas
  • 1,243
  • 6
  • 7
  • Same result as the one above, This gets rid of the '????' characters but now I'm left with lots of 'Ð'. Like 'УРА' becomes 'УРÐ' – user1425462 Oct 28 '18 at 16:52
  • That's helpful advice in general, but the problem here is one of character encoding, so there is a need to perform explicit re-encoding of the _string_ data received before further processing, so in this special case working with the `.Content` property of the `Invoke-WebRequest` response is required. – mklement0 Oct 28 '18 at 18:08
  • I was trying -ContentType "application/json; charset=utf-8" in header object for my `Invoke-RestMethod` call and it didn't work , passing it like a separate parameter to `Invoke-RestMethod -ContentType "application/json; charset=utf-8" worked perfectly – Shrikant Prabhu Oct 26 '21 at 21:21
2

Try to convert the value of the .Content property to JSON:

$request = 'http://151.80.109.18:8082/vrageremote/v1/session/players'
$a = Invoke-WebRequest -ContentType "application/json; charset=utf-8" $request

($a.Content | convertfrom-json).Data.Players | select DisplayName,FactionTag | Out-file "$scriptPath\getFactionTag.txt" -Encoding Default
mklement0
  • 382,024
  • 64
  • 607
  • 775
Peter Schneider
  • 2,879
  • 1
  • 14
  • 17