7

I'm using Invoke-RestMethod to get page names from an application I'm using. I notice that when I do a GET on the page it returns the page name like so

This page â is working

However the actual page name is

This page – is working

Here's how my request looks

 Invoke-WebRequest -Uri ("https://example.com/rest/api/content/123789") -Method Get -Headers $Credentials -ContentType "application/json; charset=utf-8"

The problem is with the en-dash, does anyone know how I can fix this?

Joel Coehoorn
  • 399,467
  • 113
  • 570
  • 794
JoeRod
  • 899
  • 8
  • 20
  • 30

3 Answers3

16

In case of Invoke-WebRequest does not detect responce encoding right, you can use RawContentStream and convert it to needed encoding:

$resp = Invoke-WebRequest -Uri ... $html=[system.Text.Encoding]::UTF8.GetString($resp.RawContentStream.ToArray());

vitrilo
  • 1,437
  • 16
  • 20
3

Invoke-restmethod or invoke-webrequest?

The Invoke-RestMethod cmdlet uses the default decoding on the result of the HttpWebResponse.CharacterSet property.

If that is not set it uses a default encoding of ISO-8859-1 by default (afaik).

I'm assuming your server is sending some wrong charset in the response headers (or dropping it) hence it's beeing decoded wrongly.

Do you know what charset/encoding are sent in your response from your server?

If you're trying the Invoke-webrequest; check your headers in your response like e.g. 

$r = invoke-webrequest http://example.com
$r.Headers

If you're dealing with an encoding issue; e.g. your server is not sending the right headers; you can always try to dump the response in a file and read it with a different encoding:

Invoke-WebRequest http://example.com -outfile .\test.txt
$content = get-content .\test.txt -Encoding utf8 -raw

In this case you will no longer be working with the http-response; but it might help you debug/find the encoding issues your looking for.

Joel Coehoorn
  • 399,467
  • 113
  • 570
  • 794
Harald F.
  • 4,505
  • 24
  • 29
  • 1
    When I run Invoke-WebRequest http://foorbar.com -outfile .\test.txt I see the correct character, but if I run the same command without the outfile the character is incorrect. – JoeRod Mar 08 '16 at 14:38
  • It's interesting that with the Invoke-RESTMethod cmdlet in PS 7, encoding works just fine. In Window PS 5.1 it doesn't. I'm no dev, but after speaking with one, the default content encoding for the `application/json` content type should be `utf-8` by default, not `ISO-8859-1` (see https://stackoverflow.com/questions/477816/what-is-the-correct-json-content-type/477819#477819). So I guess Windows PS 5.1 will never be fixed :(. I haven't yet been able to re-encode the JSON result and keep it as an array object. – Swinster Jun 21 '22 at 10:34
3

One line solution (without files):

[system.Text.Encoding]::UTF8.GetString((Invoke-WebRequest "https://www.example.com").RawContentStream.ToArray())
Joel Coehoorn
  • 399,467
  • 113
  • 570
  • 794
Gromuls
  • 31
  • 2
  • Give this man a medal !!! This is the best answer. Thank you so much :) – Pierre-Paul St-Pierre Jan 15 '21 at 00:00
  • This is close but the result ends up as being a string type, not an array as per the original output of an `Invoke-RestMethod` request. – Swinster Jun 21 '22 at 10:38
  • Ok, if you wrap the `ConvertFrom-Json` CmdLet around that, you get back to the array, so `ConvertFrom-Json -InputObject ([system.Text.Encoding]::UTF8.GetString((Invoke-WebRequest "https://www.example.com").RawContentStream.ToArray()))`. It's a bit ugly, but seems to work.... – Swinster Jun 24 '22 at 11:14
  • There does seem to be a big downside to using the `Invoke-WebRequest` and converting over the `Invoke-RestMethod`, which is speed. My script has to make multiple API calls (so iterating through a set of values and querying more detail on each of them) is now quite a lot slower. – Swinster Jun 24 '22 at 13:33