Okhttp3 request returning empty response as well as different headers than Postman

Question

I'm trying to parse the JSON from this URL, https://fantasy.premierleague.com/drf/elements but the response I get from okhttp and Postman are different. I've also used online API testers and I get a response back with the full JSON there as well. I'm not sure why my code isn't working.

Can anyone help me figure this out?

Here's the code I'm using for quick testing.

    val request = Request.Builder()
            .url("https://fantasy.premierleague.com/drf/elements")
            .get()
            .addHeader("accept", "*/*")
            .addHeader("accept-encoding", "gzip, deflate")
            .addHeader("cache-control", "no-cache")
            .addHeader("connection", "keep-alive")
            .cacheControl(CacheControl.FORCE_NETWORK)
            .build()

    val httpClient = OkHttpClient()
    httpClient.newCall(request).enqueue(object : okhttp3.Callback {
        override fun onFailure(call: okhttp3.Call?, e: IOException?) {
            Timber.d("FPL response failed = " + e?.message.toString())
        }

        override fun onResponse(call: okhttp3.Call?, response: okhttp3.Response?) {
            if (response!!.isSuccessful) {

                val responseBody = response.body()?.string()
                try {
                    val obj = JSONObject(responseBody)
                    Timber.d("FPL response = " + obj.toString())
                } catch (t: Throwable) {
                    Timber.e("Could not parse malformed JSON: " + t.message)
                }

                Timber.d("FPL response = $response")
                Timber.d("FPL headers = " + response.headers())
                Timber.d("FPL body = " + responseBody)
            } else {
                Timber.d("FPL response failed = " + response.body().toString())
            }
        }
    })

I tried replicating Postman's code snippet headers which are:

Request request = new Request.Builder()
  .url("https://fantasy.premierleague.com/drf/elements")
  .get()
  .addHeader("cache-control", "no-cache")
  .addHeader("postman-token", "05ae03ef-cf44-618c-a82c-5762e245b771")
  .build();

but sadly no luck there either.

Log:

D/HomeController$onAttach:L135: FPL response = Response{protocol=http/1.1, code=200, message=OK, url=https://fantasy.premierleague.com/drf/elements}

D/HomeController$onAttach:L138: FPL headers = 
Server: Varnish
Retry-After: 0
Content-Type: application/json
Content-Length: 0
Accept-Ranges: bytes
Date: Tue, 15 Aug 2017 22:17:18 GMT
Via: 1.1 varnish
Connection: close
X-Served-By: cache-jfk8123-JFK
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1502835438.419014,VS0,VE0

D/HomeController$onAttach:L139: FPL body =

As you can see, the body is empty and "Connection" is close.

Here's the response headers that Postman gets, much different.

Accept-Ranges →bytes
Age →14
Allow →GET, HEAD, OPTIONS
Cache-Control →no-cache, no-store, must-revalidate, max-age=0
Connection →keep-alive
Content-Encoding →gzip
Content-Language →plfplen
Content-Length →39518
Content-Type →application/json
Date →Tue, 15 Aug 2017 00:54:32 GMT
Edge-Control →max-age=60
Fastly-Debug-Digest →fdb44d2dd7c0b26c639a8b3476f8c63661c68707cc3b9446f8ed3941cd3fe01e
Server →nginx
Vary →Accept-Encoding
Via →1.1 varnish
Via →1.1 varnish
X-Cache →HIT, MISS
X-Cache-Hits →1, 0
X-Frame-Options →DENY
X-Served-By →cache-lcy1146-LCY, cache-jfk8143-JFK
X-Timer →S1502758472.116470,VS0,VE82

as you can see, "Connection" says keep alive.

Is there something I'm missing in my Request.Builder() to get this to work?

EDIT:

So, I decided to try making the request using an AsyncTask, and got the full JSON back as you can see in the log under. I don't understand why Okhttp3 isn't working.

            val url = params[0]
            var stream: InputStream? = null
            var connection: HttpsURLConnection? = null
            var result: String? = null
            try {
                connection = url?.openConnection() as HttpsURLConnection?
                // Timeout for reading InputStream arbitrarily set to 3000ms.
                connection?.readTimeout = 3000
                // Timeout for connection.connect() arbitrarily set to 3000ms.
                connection?.connectTimeout = 3000
                // For this use case, set HTTP method to GET.
                connection?.requestMethod = "GET"
                // Already true by default but setting just in case; needs to be true since this request
                // is carrying an input (response) body.
                connection?.doInput = true
                // Open communications link (network traffic occurs here).
                connection?.connect()

                val responseCode = connection?.responseCode

                if (responseCode != HttpsURLConnection.HTTP_OK) {
                    throw IOException("HTTP error code: " + responseCode)
                }
                // Retrieve the response body as an InputStream.
                stream = connection?.inputStream

                Timber.d("httpurl stream = " + connection?.inputStream.toString())

                if (stream != null) {
                    // Converts Stream to String with max length of 500.
                result = readStream(stream, 500)
                }
            } finally {
                // Close Stream and disconnect HTTPS connection.
                if (stream != null) {
                    stream.close()
                }
                if (connection != null) {
                    connection.disconnect()
                }
            }

            return result

Log:

 D/HomeController$NetworkA:L266: httpurl response = [{"id":1,"photo":"48844.jpg","web_name":"Ospina","team_code":3,"status":"a","code":48844,"first_name":"David","second_name":"Ospina","squad_number":13,"news":"","now_cost":50, .... }]

That URL redirects with a 301. Try adding a slash on the end (`https://fantasy.premierleague.com/drf/elements/`) to get to the redirected URL. OkHttp is supposed to follow redirects automatically, so this may not help. But, testing with `curl` (which does not follow redirects), I get the same results as you do with OkHttp using your original URL, and I get the JSON response if I add the `/` at the end. — CommonsWare, Aug 15 '17 at 22:33
@CommonsWare Ok, just tried adding `/` to the end but still no luck. I tried the url with `curl` as well and it does return the response. I'm not sure why this isn't working using Okhttp3. — MikeOscarEcho, Aug 15 '17 at 23:49
The only other thing that I can think of is that perhaps OkHttp does not normalize HTTP header capitalization, and perhaps the server isn't liking your all-lowercase header names. — CommonsWare, Aug 15 '17 at 23:53
@CommonsWare See my edit, I just made the same request using the old fashioned way of using an `AsyncTask`, `HttpsURLConnection`, and `InputStreamReader` and I got the full JSON back. Really odd that `Okhttp3` isn't working. — MikeOscarEcho, Aug 16 '17 at 00:25
Might be worth putting together a reproducible test case and post it as an issue. If HURL can handle it, OkHttp should. — CommonsWare, Aug 16 '17 at 00:54
A way to help debug this is to add a NetworkInterceptor that logs the HTTP requests so you can dive into what is actually happening. Here is a runnable example configuring [OkHttpClient logging interceptor](https://www.stubbornjava.com/posts/okhttpclient-logging-configuration-with-interceptors#okhttp-interceptors) This will give you much better debug logging as well as all the network hops between the first request and the final response. You can probably use Timber logging instead of SLF4J — Bill O'Neil, Aug 16 '17 at 02:32

Abhijit Sarkar · Accepted Answer · 2017-08-16T05:36:16.587

Your problem is User-Agent header. Here is a working example:

fun syncGetOkHttp() {
    println("\n===")
    println("OkHttp")
    println("===")
    val client = OkHttpClient().newBuilder()
            .addNetworkInterceptor { chain ->
                val (request, response) = chain.request().let {
                    Pair(it, chain.proceed(it))
                }

                println("--> ${RequestLine.get(request, Proxy.Type.HTTP)})")
                println("Headers: (${request.headers().size()})")
                request.headers().toMultimap().forEach { k, v -> println("$k : $v") }

                println("<-- ${response.code()} (${request.url()})")

                val body = if (response.body() != null)
                    GZIPInputStream(response.body()!!.byteStream()).use {
                        it.readBytes(50000)
                    } else null

                println("Response: ${StatusLine.get(response)}")
                println("Length: (${body?.size ?: 0})")

                println("""Body: ${if (body != null && body.isNotEmpty()) String(body) else "(empty)"}""")
                println("Headers: (${response.headers().size()})")
                response.headers().toMultimap().forEach { k, v -> println("$k : $v") }

                response
            }
            .build()

    Request.Builder()
            .url(url)
            .header("Accept", "application/json")
            .header("User-Agent", "Mozilla/5.0")
            .build()
            .let { client.newCall(it).execute() }
}

Yea, that was it! Do you happen to know why those are required? Wouldn't mind learning a bit more. Thanks for the help. — MikeOscarEcho, Aug 17 '17 at 02:24
@MikeOscarEcho upto the server; perhaps it wants to set a cookie. — Abhijit Sarkar, Aug 17 '17 at 03:31

Okhttp3 request returning empty response as well as different headers than Postman

1 Answers1

Linked