How to escape non-ascii symbols in PowerShell 5.1?

Question

I'm trying to send a file content to server:

$uri = ...
$headers = @{
    ...
    "Content-Type" = "application/json"
}
[string] $content = Get-Content .\filename -Encoding utf8 -Raw
$body = @{
    ...
    "content" = $content
} | ConvertTo-Json
$response = Invoke-WebRequest $uri -Method 'PUT' -Headers $headers -Body $body

But all of non-ascii symbols are changed to another similar symbols or question marks

How can I escape them?

I've read documentation and I know about parameter -EscapeHandling of cmdlet ConvertTo-Json, but it's available from PowerShell 6.2, I have only 5.1

Can you try `"Content-Type" = "application/json; charset=utf-8"` ? — Theo, Nov 07 '20 at 12:44
(a) Does `$content` look correct locally? (b) What encoding does the response indicate as being returned? `$response.Headers['content-type']` — mklement0, Nov 07 '20 at 13:37
@mklement0, Yes, it looks correctly locally. I see cyrillic symbols as they are. And after conversion only control symbols are escaped. [Screenshot](https://i.stack.imgur.com/18xRR.png). `$response.Headers['content-type'] == "application/json"` — Андрей NOP, Nov 07 '20 at 16:00
@mklement0 Hm... This script runs in TeamCity and I see question marks there instead of cyrillic symbols... [Screenshot](https://i.stack.imgur.com/UeoFw.png) — Андрей NOP, Nov 07 '20 at 16:04
I know nothing about TeamCity, unfortunately, but perhaps you must configure its console to use UTF-8. — mklement0, Nov 07 '20 at 18:08
@mklement0, actually I'm not interested in console output. I just need to send the file content. Is it related? Anyway I should try to do this request from my pc. — Андрей NOP, Nov 07 '20 at 18:55
No, for correct data handling (that doesn't involve _external programs_), the console shouldn't matter. Do note that `"application/json"` as the content type - without an explicit `charset` field - makes `Invoke-WebRequest` assume ISO-8859-1 encoding, not UTF-8, so if the data coming back is in actually UTF-8 encoded, you'll have to reencode. — mklement0, Nov 07 '20 at 19:09
For reencoding, see [this answer](https://stackoverflow.com/a/47961370/45375) — mklement0, Nov 07 '20 at 19:13

score 0 · Accepted Answer · answered Nov 09 '20 at 06:59

As a result, I wrote a simple function:

function EscapeNonAscii([string] $s)
{
    $sb = New-Object System.Text.StringBuilder;
    for ([int] $i = 0; $i -lt $s.Length; $i++)
    {
        $c = $s[$i];
        if ($c -gt 127)
        {
            $sb = $sb.Append("\u").Append(([int] $c).ToString("X").PadLeft(4, "0"));
        }
        else
        {
            $sb = $sb.Append($c);
        }
    }
    return $sb.ToString();
}

And used it like this:

$updateFileResponse = Invoke-WebRequest $updateFileUri -Method 'PUT' -Headers $headers -Body (EscapeNonAscii $body)

It helped. For everybody who will google it in the future, it's a request to GitLab API Update existing file in repository

PS: I use PS as C# because I know it badly. If somebody knows how to rewrite this fragment better please let me know.

PPS: And also I know that StringBuilder.Append changes an existing object, but I add here assigning ($sb = $sb.Append($c) instead of simple $sb.Append($c)) because it prints every action to console. If you know how to fix it please let me know.

How to escape non-ascii symbols in PowerShell 5.1?

1 Answers1