How should I write an URLFetch to avoid timeout exceptions? GAE/J + Facebook app

Question

The function is very simple:

public static String readURL(URL url, HTTPMethod method) throws IOException {
    FetchOptions opt = FetchOptions.Builder.doNotValidateCertificate();
    HTTPRequest request = new HTTPRequest (url, method, opt);
    URLFetchService service = URLFetchServiceFactory.getURLFetchService();
    HTTPResponse response = service.fetch(request);
    byte[] content = response.getContent();
    return new String(content);
}

The problem is that occasionally I get an exception:

java.lang.RuntimeException: java.io.IOException: Timeout while fetching: https://graph.facebook.com...

What I would like to do is to the function in some while() loop or similar so it keeps trying until the response is there. What do you think would be the best approach? Am I heading to the right direction or would you suggest something totally different? Increasing the timeout to 10 secs as was suggested in some other posts should avoid the majority of the issues, but not eradicate the problem.

Thanks.

PS 1: The line FetchOptions opt = FetchOptions.Builder.doNotValidateCertificate(); is needed to avoid another problem,

HTTP ERROR 500
Problem accessing /auth. Reason:
javax.net.ssl.SSLHandshakeException: Could not verify SSL certificate for: https://graph.facebook.com/oauth/access_token?...

PS 2: This is not an issue like in this thread: GoogleAppEngine urlfetch timeout exception because I'm fetching the facebook servers, not my own server. Nor is like "Timeout while fetching" URLFetch GAE/J because the issue is not a very large feed, but an non-responsive server (specially slow when responding requests by test users)

regarding the SSL issue with facebook, if it's something you hit in the latest 1.4.2. download the SDK(manually not with eclipse) see http://code.google.com/p/googleappengine/issues/detail?id=4603 — maximbr, Mar 19 '11 at 16:49
I was aware it was a bug in 1.4.2 and I was using that as a workaround. I didn't know it was already solved. Thanks for the tip. — Aleadam, Mar 19 '11 at 17:21

systempuntoout · Accepted Answer · 2011-03-19T17:34:13.073

First of all, increase the urlfetch timeout to 10 seconds, it will make a huge difference in term of fewer raised Timeout exceptions.

I would try to download the facebook data from the Web handler the first time and, in case of TimeOut exception, I would move the task to a TaskQueue.

Once in a TaskQueue, if a TimeOut exception is raised and the Task fails, App Engine automatically retries it, trying to download the content of the Url until it succeeds.

import com.google.appengine.api.taskqueue.Queue;
import com.google.appengine.api.taskqueue.QueueFactory;
import static com.google.appengine.api.taskqueue.TaskOptions.Builder.*;

        Queue downloadqueue = QueueFactory.getDefaultQueue();
        queue.add(withUrl("/worker").param("url", urlToFetch))

Much more elegant solution than a while() loop. I'll dig deeper into TaskQeue then. Thanks. — Aleadam, Mar 19 '11 at 17:20

How should I write an URLFetch to avoid timeout exceptions? GAE/J + Facebook app

1 Answers1