Google App Engine - HttpClient execute method closes connection without getting data

Question

The Problem

I am using Apache HttpClient in a web servlet hosted on google app engine. The problem i have is that if i visit the url in a web browser, i get a good JSON result, but if i try to get this same JSON data in JAVA my connection is immediatly closed with no returned entity. Here is the code from my servlet:

try {
    // Pooling code from: http://stackoverflow.com/a/13763608
    PoolingClientConnectionManager cxMgr = new PoolingClientConnectionManager(
            SchemeRegistryFactory.createDefault());
    cxMgr.setMaxTotal(100);
    cxMgr.setDefaultMaxPerRoute(20);

    HttpClient client = new DefaultHttpClient(cxMgr);
    URIBuilder uriQuery = new URIBuilder();

    uriQuery.setScheme("https");
    uriQuery.setHost("doctorbase.com");
    uriQuery.setPath("/api/dr/search_rest");
    uriQuery.setParameter("location", user.getLocation().getName());
    URI uri = uriQuery.build();

    System.out.println("Setting up get: " + uri.toString());
    HttpGet getRequest = new HttpGet(uri);
    System.out.println("Calling execute");
    HttpResponse docApiResponse = client.execute(getRequest);
    System.out.println("Getting Entity");
    HttpEntity entity = docApiResponse.getEntity();
} catch (Exception e) {
    System.out.println("Exception getting doctors: "
            + e.toString());
    e.printStackTrace();
}

And here is the log i see in Google App Engine (it appears that the client.Execute(getRequest) line is popping out of the method which shouldnt happen since i have it in a try-catch block and the catch is not executed:

2013-03-04 19:44:54.593 [s~mybloodpressurelog/1.365731215144292261].<stdout>: Calling getNearbyDoctorInfo
I 2013-03-04 19:44:54.747 [s~mybloodpressurelog/1.365731215144292261].<stdout>: Setting up get: https://doctorbase.com/api/dr/search_rest?location=Sacramento%2C+California
I 2013-03-04 19:44:54.747 [s~mybloodpressurelog/1.365731215144292261].<stdout>: Calling execute
D 2013-03-04 19:44:54.752 org.apache.http.impl.conn.PoolingClientConnectionManager requestConnection: Connection request: [route: {s}->https://doctorbase.com][total kept alive:
D 2013-03-04 19:44:54.753 org.apache.http.impl.conn.PoolingClientConnectionManager leaseConnection: Connection leased: [id: 1][route: {s}->https://doctorbase.com][total kept al
D 2013-03-04 19:44:54.758 org.apache.http.impl.conn.DefaultClientConnection shutdown: Connection org.apache.http.impl.conn.DefaultClientConnection@f0850b shut down
D 2013-03-04 19:44:54.758 org.apache.http.impl.conn.DefaultClientConnection close: Connection org.apache.http.impl.conn.DefaultClientConnection@f0850b closed
D 2013-03-04 19:44:54.758 org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection: Connection released: [id: 1][route: {s}->https://doctorbase.com][total kep

Any help solving the mystery is the immediately closing connection is appreciated.

Working code

(thanks Fahad for pointing me in the right direction!):

URL doctorUrl = new URL(
    "https://doctorbase.com/api/dr/search_rest?"
            + user.getLocation().getName()
                    .replaceAll(" ", "%2C+"));
try {
    System.out.println("IS to String");
    BufferedReader jsonReader = new BufferedReader(
            new InputStreamReader(doctorUrl.openStream()));
    String json = jsonReader.readLine();

    System.out.println("Doctor JSON: " + json);

    DoctorApiResult drs = new Gson().fromJson(json,
            DoctorApiResult.class);

    System.out.println("Deserialized Doctors ");

    int numberOfDoctors = drs.getDrs().size();
    System.out.println("Number of doctors found: "
            + numberOfDoctors);

    DoctorBean[] doctors = new DoctorBean[numberOfDoctors];
    drs.getDrs().toArray(doctors);

    request.setAttribute("drs", doctors);
} catch (Exception e) {
    System.out
            .println("Exception getting doctors: " + e.toString());
    e.printStackTrace();
    request.setAttribute("error", e.toString());
}

Could it be the `User-Agent` header HttpClient is sending to the server? Browsers' may be known user-agent to the server but Apache HttpClient's user-agent is not allowed? — Bimalesh Jha, Mar 05 '13 at 04:05
@BimaleshJha: Excellent suggestion, unfortunantly it did not work :\ I tired the user-agent strings for the latest versions of Firefox and Chrome (from here: http://www.useragentstring.com/pages/useragentstring.php ) and neither got me back data. — Steven Magana-Zook, Mar 05 '13 at 04:24

score 3 · Accepted Answer · answered Mar 05 '13 at 05:07

3

In Google AppEngine those library are blacklisted use URLFetchService in google-http-java-client to connect external servers from GAE

References

https://code.google.com/p/google-http-java-client/

https://developers.google.com/appengine/docs/java/javadoc/com/google/appengine/api/urlfetch/URLFetchService

answered Mar 05 '13 at 05:07

Fahad

749
6
12

Thank you this worked great! I posted the code that I used to make it work in the question above (I used the method from here: https://developers.google.com/appengine/docs/java/urlfetch/usingjavanet ) – Steven Magana-Zook Mar 05 '13 at 05:37

Google App Engine - HttpClient execute method closes connection without getting data

The Problem

Working code

1 Answers1