How to solve "UTF-8" is not supported encoding name

Question

I am using HttpClient to send a post request to a remote server where I do not have control and trying to get the response as following

HttpResponseArgs result = //Make the request with HttpClient object ( I am skipping it here)

var stringResult = result?.Content.ReadAsStringAsync().Result; // exception thrown here as "UTF-8" is not supported encoding name

on debug I found the Content-Type header in response from remote server is set as "text/xml; charset ="UTF-8"" Please note the extra "" between UTF-8 this is causing the error if I remove the header from response and put a new content-Type header with "text/xml; charset =UTF-8". please note I removed the extra Quote around UTF-8 the code

result?.Content.ReadAsStringAsync().Result; // works fine now

Please suggest what can I do? I feel its a bug in .net framework as postman can interpret response of remote server in correct way.

the problem is in double quoate around UTF-8 in header of response

Don't use `.Result` to make async code work, you will lock up the program. — Neil, May 11 '22 at 13:08
https://www.rfc-editor.org/rfc/rfc7231.html#section-3.1.1.5 explicitly allow `Text/HTML;Charset="utf-8"` so I'm surprised that .net doesn't support that format. — Neil, May 11 '22 at 13:13
Open the raw stream, the wrap with your own `StreamReader`? https://github.com/dotnet/runtime/issues/42079 Should be fixed since .net core 3. — Jeremy Lakeman, May 16 '22 at 06:30

score 2 · Accepted Answer · answered May 22 '22 at 19:52

You can use custom EncodingProvider

public class Utf8EncodingProvider : EncodingProvider
{
    public override Encoding GetEncoding(string name)
    {
        return name == "\"UTF-8\"" ? Encoding.UTF8 : null;
    }

    public override Encoding GetEncoding(int codepage)
    {
        return null;
    }

    public static void Register()
    {
        Encoding.RegisterProvider(new Utf8EncodingProvider());
    }
}

Usage based on your question code:

Utf8EncodingProvider.Register(); //You should call it once at startup of your application
HttpResponseArgs result = //Make the request with HttpClient object ( I am skipping it here)

var stringResult = result?.Content.ReadAsStringAsync().Result; // Must be executed without exception.

It worked and is also the proposed solution in Github issue page hence marked as correct answer — user3048027, Jun 02 '22 at 07:01

score 1 · Answer 2 · answered May 19 '22 at 13:12

1

Perhaps force UTF-8 over the raw bytes, something like this

var buffer = await response.Content.ReadAsBufferAsync();
var byteArray = buffer.ToArray();
var responseString = Encoding.UTF8.GetString(byteArray, 0, byteArray.Length);

answered May 19 '22 at 13:12

Yarin_007

1,449
1
10
17

score 0 · Answer 3 · answered May 23 '22 at 03:15

There are many ASCII incompatible encodings in widespread use, particularly in Asian countries (which had to devise their own solutions before the rise of Unicode) and on platforms such as Windows, Java and the .NET CLR, where many APIs accept text as UTF-16 encoded data.

You could...

Pass it to python as a fileobject, use the encoding modules to change it back and forth. this technique has secured python as a "glue" language for C++ people.

Once again, just my opinion.

but i would strongly suggest you take Georgy Tarasovs answer

score 0 · Answer 4 · answered Aug 30 '23 at 15:10

A modification of @Georgy Tarasov was necessary for me, because a page kept failing with utf8 as it sent a slight variation.

public class Utf8EncodingProvider : EncodingProvider
{
    private static readonly HashSet<string> Utf8Encoders = new HashSet<string>(
        new string[] { "utf-8", "utf8", "\"UTF-8\"" }
    );

    public override Encoding GetEncoding(string name)
    {
        if (Utf8Encoders.Contains(name))
        {
            return Encoding.UTF8;
        }

        return null;
    }

    public override Encoding GetEncoding(int codepage)
    {
        return null;
    }

    public static void Register()
    {
        Encoding.RegisterProvider(new Utf8EncodingProvider());
    }
}

Don't forget to call

Utf8EncodingProvider.Register();

One could convert name.toLower() for more hits

How to solve "UTF-8" is not supported encoding name

4 Answers4