1

Upon executing Confluence REST API calls I get back a response encoded in UTF-8. However, when I export the results with either Out-File or Export-CSV even with the -Encoding utf8 parameter German Umlauts are not correctly represented. For example, 'ü' is still 'ü'.

From what I could gather it's due to the fact that PowerShell 5.1 natively relies on Windows-1252. I verified that Umlauts are preserved when using PowerShell Core by executing

[psobject].Assembly.GetTypes() | Where-Object { $_.Name -eq 'ClrFacade'} | ForEach-Object { $_.GetMethod('GetDefaultEncoding', [System.Reflection.BindingFlags]'nonpublic,static').Invoke($null, @()) }

Even changing the script file itself to use the encoding UTF-8 with BOM or Windows-1252 does not preserve Umlauts, neither in the PowerShell nor exportet output.

Do you know of any way to tell PowerShell 5.1 to preserve Umlauts while executing the REST call?

I cannot use PowerShell core as further operations require cmdlets which do net yet exist for PowerShell Core.

Thanks!

colonel_claypoo
  • 543
  • 1
  • 12
  • 25
  • Can you try with the `-Outfile` parameter? – AdminOfThings Oct 24 '19 at 10:51
  • Do you know which part of your workflow is breaking the encoding? e.g. is it definitely invoke-restmethod, or maybe out-file or the viewer you're checking the file contents in? for example if i do this ```"ü" | out-file c:\temp\umlaut-utf8.txt -encoding utf8``` i get a file with the bytes "EF BB BF C3 BC 0D 0A" which shows "ü" in notepad, but "ü" in a hex editor. – mclayton Oct 24 '19 at 10:52
  • @AdminOfThings, what you suggested does indeed preserve the Umlauts, that's nice. However, I noticed that it only does so when specifying the .txt extension. .csv does not work here as well. Ultimately, I would want to store REST response details in a CSV-file. – colonel_claypoo Oct 24 '19 at 11:07
  • @mclayton, I suspect it's the shell, i.e. PowerShell 5.1. in VSCode (but also powershell.exe in standalone mode behaves the same). I guess `"ü" | out-file` works because it's not first "internalized" by the shell as when executing a cmdlet. So to generalize, each Umlaut that is processed by a cmdlet in PS 5.1. should exhibit the encoding issue. – colonel_claypoo Oct 24 '19 at 11:11
  • do you have a simple repro i can try here? i tried this in a powershell.exe console, but the file shows the umlaut fine in notepad: ```$xmldoc = Invoke-RestMethod -Uri "https://en.wikipedia.org/wiki/%C3%9C"; $xmldoc.DocumentElement.InnerText | out-file c:\temp\out.txt -Encoding utf8;``` – mclayton Oct 24 '19 at 11:26
  • @mclayton, yes, that works just fine for some reason. I dug a little deeper, it might be a Confluence pecularity. The response does not return a charset in the Content-Type; see https://jira.atlassian.com/browse/CONFSERVER-56064 I also just verified with Fiddler that executing the pure REST call does indeed not return a charset, too bad. – colonel_claypoo Oct 24 '19 at 11:55
  • In that case, you can possibly pull the raw bytes from the response and decode them yourself using the appropriate content type. I can post an example in a bit, but just search for “decode bytes utf8 .net” or similar and you might find an answer... – mclayton Oct 24 '19 at 12:24
  • I believe I have found the answer here https://stackoverflow.com/questions/55265157/how-do-i-change-my-powershell-script-so-that-it-writes-out-file-in-ansi-window. I don't quite understand it yet but simply adding the `-Encoding default` parameter fixes the issue. – colonel_claypoo Oct 24 '19 at 12:25

1 Answers1

1

As discussed in the comments, it looks like the Confluence API encodes http responses using UTF8, but does not include the "Content-Type" header to indicate that.

The HTTP specification for the charset parameter says that in the absence of this header, the client should assume it's encoded with ISO-8859-1 character set, so what is happening in your request is something like this:

# server (Confluence API) encodes response text using utf8
PS> $text = "ü";
PS> $bytes = [System.Text.Encoding]::UTF8.GetBytes($text);
PS> write-host $bytes;
195 188

# client (Invoke-RestMethod) decodes bytes as ISO-8859-1
PS> $text = [System.Text.Encoding]::GetEncoding("ISO-8859-1").GetString($bytes);
PS> write-host $text;
ü

Given that you can't control what the server sends, you'll either need to capture the raw bytes yourself (e.g. using System.Net.Http.HttpClient) and decode them using UTF8, or modify the existing response to compensate for the encoding mismatch (e.g. below).

PS> $text = "ü"
PS> $bytes = [System.Text.Encoding]::GetEncoding("ISO-8859-1").GetBytes($text)
PS> $text = [System.Text.Encoding]::UTF8.GetString($bytes)
PS> write-host $text
ü

Note that if you use the -Outfile parameter of Invoke-RestMethod it presumably streams the response bytes directly to disk without decoding or encoding them, so the resultant file already contains utf8 $bytes rather than utf8 $bytes -> string decoded using ISO-8859-1 -> file bytes encoded using utf8

mclayton
  • 8,025
  • 2
  • 21
  • 26