"Unexpected character exception" when making GET request to StackExchange API using Flurl

Question

I have created a console application in which I'm making a simple GET request to the Stack Exchange API to fetch some comments. I'm using Flurl. This method is called from Main

private static async Task GetComments()
{
    dynamic d = await "https://api.stackexchange.com/2.2/comments?page=1&pagesize=5&order=desc&min=1513468800&max=1513555200&sort=creation&site=stackoverflow"
                        .GetJsonAsync();
}

But, I get this error:

{"Unexpected character encountered while parsing value: \u001f. Path '', line 0, position 0."}

I have tried setting the headers like this with no luck.

dynamic d = await new Url("https://api.stackexchange.com/2.2/comments.....")
               .WithHeader("Content-Encoding", "gzip")
               .WithHeader("Accept-Encoding", "gzip")
               .GetJsonAsync();

The URL does return proper JSON when I open it in the browser

Maybe flurl is not as forgiving as a browser and you need to remove the extra slash in the url. — Crowcoder, Dec 18 '17 at 18:21
@Crowcoder It should. I'm basically trying to do [this](https://stackoverflow.com/questions/27108264/c-sharp-how-to-properly-make-a-http-web-get-request), but with Flurl — adiga, Dec 18 '17 at 18:28
@DavidG I don't know. Couldn't find anything in the documentation related to that. — adiga, Dec 18 '17 at 18:32
Maybe you have some invisible character in url which confuses flurl? What if type the whole url by hand (if you copy pasted it from somewhere)? — Evk, Dec 18 '17 at 18:32
"It should" is not an acceptable answer. Did you try `.WithHeader("Accept-Encoding", "identity")` (i.e. request that `gzip` isn't used)? — ikegami, Dec 18 '17 at 18:34
The API result is gzipped whether you request it or not. You need to decompress it before calling anything like `GetJsonAsync`. — Brock Adams, Dec 18 '17 at 18:37
It works for me with `HttpClient`. No headers needed. Why flurl? — Crowcoder, Dec 18 '17 at 18:37
@ikegami didn't work with `.WithHeader("Accept-Encoding", "identity")` — adiga, Dec 18 '17 at 18:38
@ikegami, the SE API will not honor such a request.You get gzip unless you request deflate. Those are your only options with the SE API. — Brock Adams, Dec 18 '17 at 18:39
@Crowcoder I have just started learning Flurl. I thought I'd use the StackExchange API. — adiga, Dec 18 '17 at 18:40
@Crowcoder If this worked for you with HttpClient without any special config, I would bet anything the OP's original Flurl snippet would work for you too. It did for me. I think that's because the API sent the content DEFLATE encoded, which HttpClient (and Flurl) can decompress automatically by default. I don't know if the API randomly picks GZIP or DEFLATE if you don't specify or what, but they [recommend](https://api.stackexchange.com/docs/compression) specifying one or the other. — Todd Menier, Dec 20 '17 at 02:49
@adiga Just a head up, I'm going to make major edits to my answer because I was dead wrong about HttpClient and Flurl supporting DEFLATE by default. Neither do. It was a case of being on little sleep and losing track of what I had done in my debugging process ;) — Todd Menier, Dec 20 '17 at 14:57
@Crowcoder I was wrong about DEFLATE, but I stand by Flurl's defaults matching whatever HttpClient's are. I suspect that whether they both work or both don't comes down to [differences between platforms/versions](https://github.com/dotnet/platform-compat/issues/22). — Todd Menier, Dec 20 '17 at 15:10
Jon Skeet actually [proved](https://github.com/google/google-api-dotnet-client/issues/1110) the differences and linked to where it's [documented](https://learn.microsoft.com/en-gb/dotnet/api/system.net.http.httpclienthandler.automaticdecompression). And who's to argue with Jon Skeet? ;) — Todd Menier, Dec 20 '17 at 15:59

score 3 · Accepted Answer · answered Dec 18 '17 at 19:13

So it seems the Flurl doesn't support Gzip out of the box and getting it to work takes a bit of massaging. First you need a custom Http Client factory:

public class GzipClientFactory : Flurl.Http.Configuration.IHttpClientFactory
{
    public HttpMessageHandler CreateMessageHandler() => new HttpClientHandler()
    {
        AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
    };

    public HttpClient CreateHttpClient(HttpMessageHandler handler) => 
        new HttpClient(handler);
}

Now configure Flurl to use this:

FlurlHttp.Configure(settings => {
    settings.HttpClientFactory = new GzipClientFactory();
});

And now Gzip compression will be supported:

dynamic d = await new Url("https://api.stackexchange.com/2.2/comments.....")
               .GetJsonAsync();

One little change I'd suggest is inherit from `DefaultHttpClientFactory` and only override `CreateMessageHandler`. It would trim a little off the implementation. — Todd Menier, Dec 20 '17 at 02:39

Todd Menier · Answer 2 · 2018-02-22T13:47:51.963

UPDATE: Flurl.Http now supports automatic decompression by default, so just upgrade and you can avoid all of this.

Here's an option that resembles the accepted answer, except it only applies to calls to api.stackexchange.com, and it's more or less a one-liner. Call it once at startup:

FlurlHttp.ConfigureClient("https://api.github.com", cli => 
    ((HttpClientHandler)cli.HttpMessageHandler).AutomaticDecompression =
        DecompressionMethods.GZip | DecompressionMethods.Deflate);

As a side-note, Flurl's defaults are exactly the same as HttpClient's defaults in terms of automatic decompression. There are notes in the comment that GZIP works with HttpClient, which had me scratching my head, but it seems that with HttpClient there are differences depending on platform/version. To make things easier and more predictable, I am considering supporting GZIP/DEFLATE by default in Flurl, but I first need to understand the implications in terms of performance, etc.

Okay, I'll try this as well. The answers in [this question](https://stackoverflow.com/questions/27108264/c-sharp-how-to-properly-make-a-http-web-get-request) set the `DecompressionMethods` explicitly for `HttpWebRequest` and `HttpClient`. But again, the question is 3 years old. — adiga, Dec 20 '17 at 15:38
`HttpWebRequest` is the predecessor to `HttpClient` (in practical terms). Flurl uses `HttpClient` under the hood, and this code effectively does the same as the `HttpClient` answer in the question you linked to - sets the decompression methods on the `HttpClientHandler`. — Todd Menier, Dec 20 '17 at 15:56

score 1 · Answer 3 · answered Feb 22 '18 at 13:44

1

Flurl.Http 2.2 was just released, and it will now automatically decompress GZIP and DEFLATE by default. So, the new best answer is: upgrade. :)

answered Feb 22 '18 at 13:44

Todd Menier

37,557
17
150
173

It it possible to demonstrate how that works now? I think that would be helpful to future readers. – Filnor Feb 22 '18 at 13:53
There's quite literally nothing to demonstrate. The OP's first code sample will just work as-is. No error, nothing to configure. – Todd Menier Feb 22 '18 at 15:46

score 0 · Answer 4 · answered Sep 18 '19 at 09:58

This is 2019 and i couldn't find a proper library for dotnet to consume the API's. So i created one myself. Still a long way to go. I am planning to maintaining it as long as i can.It have published it as nuget as well as you can have a look at the source here on Github

From Nuget

Install-Package StackExchange.NET -Version 1.1.0

"Unexpected character exception" when making GET request to StackExchange API using Flurl

4 Answers4