4

I'm having problems getting data from the Twitter search API with Google App Engine. Working in the personal development environment I have no problems. I'm able to get the JSON I want. However, once the code gets deployed to App Engine, I encounter a HTTP 500 Error Code.

The only ideas I have for whats going on is that there is some fundamental difference between the dev and live App Engine environments that I'm missing and/or that Twitter is refusing requests from the App Engine IP I'm running from. I wrote a small @app.route to absolutely verify that there are issues - is is below.

I'm pretty sure I'm not hitting a rate limit because the Twitter Search API doesn't explicitly have one. I saw that they requested a user-agent so I've been providing that but I'm still getting the same 500 Error.

The live version of what I'm working with is here: http://1-alpha-3.rich90usa.appspot.com/twitter_test

Any thoughts on what's going wrong here would be greatly appreciated.

Relevant Section from Flask:

@app.route('/twitter_test')
def twitter_test():
  twitter_geo_url = 'http://search.twitter.com/search.json?callback=?&rpp=100&geocode=29.6516344,-82.3248262,3mi'
  twitter_result = urlfetch.fetch(twitter_geo_url, fetch_headers)
  if twitter_result.status_code == 200:
    return twitter_result.content

Where 'fetch_headers' is previously defined as:

fetch_headers = {'User-Agent': "APPNAME-1-alpha-3"}

Top of my Flask py file:

from flask import Flask, request, render_template, session, redirect
import oauth2 as oauth
import simplejson as json
import urlparse
import urllib
import httplib
import time

from google.appengine.api import urlfetch
Richard Brooks
  • 43
  • 2
  • Need error message + stack trace. – hyperslug Apr 28 '11 at 13:23
  • All App Engine requests come from the same IP pool. It could be that Twitter is getting so many search requests from App Engine IPs that they are blocking further queries. I believe this has happened before on GAE. – Calvin Apr 28 '11 at 16:14
  • 2
    Related: http://stackoverflow.com/questions/4821888/app-engine-alternatives-to-urlfetch-seems-very-unreliable and http://stackoverflow.com/questions/4722315/cron-works-on-local-host-but-not-when-deployed-appengine – systempuntoout Apr 29 '11 at 07:49

1 Answers1

1

As Calvin indicated, all App Engine apps source outbound HTTP requests from the same pool of IPs, so the issue is almost certainly rate limiting:

From http://dev.twitter.com/pages/rate-limiting:

However, all requests coming from an IP address are applied to a Search Rate Limit. The Search Rate Limit isn't made public to discourage unnecessary search usage and abuse, but it is higher than the REST Rate Limit. We feel the Search Rate Limit is both liberal and sufficient for most applications and know that many application vendors have found it suitable for their needs.

They also call out GAE specifically as being ineligible for whitelisting:

An application must have a static IP to be eligible for whitelisting. This means the majority of cloud platforms, including Google App Engine, cannot be whitelisted.

Twitter's suggestion to get around rate-limiting is to switch to the Streaming API.

Drew Sears
  • 12,812
  • 1
  • 32
  • 41