19

Are there any guidelines to writing Google App Engine Python code that would work without Google's infrastructure on other platforms?

Is there any known attempt to create an open source framework which can run applications designed for Google App Engine on other platforms?

Edit:

To clarify, the question really is:

If I develop an application on Google App Engine now, will I be able to migrate to another platform later, or is it a lock in?

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Alterlife
  • 6,557
  • 7
  • 36
  • 49
  • 1
    I remember reading once about some guy that installed the SDK on his own server and used it to serve GAE applications. I don't remember the page though and anyway it was just some kind of experiment. – Ionuț G. Stan May 21 '09 at 11:17

9 Answers9

33

There's a number of components required to make an app fully portable:

  • The runtime environment itself. This can be ported relatively simply, by setting up a CGI or FastCGI server that emulates the App Engine environment (which itself is basically slightly-enhanced CGI). Most of the code to do this is already in the SDK. Unfortunately, there's no easy pre-packaged toolkit for this yet.
  • The datastore. By far the most complicated API to port. There are a number of efforts underway: AppScale runs on EC2/Eucalyptus/Xen and uses a HyperTable or HBase backend; it's still very much beta quality and they don't distribute the data connector separately (it's the beginnings of a complete run-your-app-on-your-own-cloud solution). Jens is/was writing an SQLite backend, and there's my own project, BDBDatastore, which uses BDB-JE as the backend, and is fully functional (though beta quality). AppDrop, which others have mentioned, simply uses the development server as a backend, and hence isn't suitable for production use.
  • The users API needs replacing with something else, such as an OpenID based API. Again, fairly simple, but no premade solutions yet.
  • The Memcache API needs a backend that uses standard C memcache backends.
  • The other APIs have perfectly functional backends as part of the SDK, so don't really need porting.
  • Cron support would also need implementing, as would background processing, XMPP, etc, when they become available.

As you can see, there's a lot of work to be done, but no fundamental barriers to making your App Engine app run outside Google's environment. In fact, if you're interested, you're more than welcome to participate - I and others have plans to combine the solutions for the various pieces into a single 'OpenEngine' solution for hosting your own apps.

Nick Johnson
  • 100,655
  • 16
  • 128
  • 198
  • 2
    +1, excellent and detailed analysis plus useful pointers. – Alex Martelli May 22 '09 at 01:38
  • In Summary: No solution exists now, but a lot of people are working toward it :) . Thank you. – Alterlife May 28 '09 at 05:04
  • Has there been any appreciable progress on this effort since May of 09? – Luke Bayes Aug 12 '10 at 21:20
  • I haven't heard much from the AppScale folks lately, though I presume they're still slogging on. TyphoonAE is making amazing progress, and the author, Tobias, does an excellent job of keeping it up to date with the latest App Engine innovations. – Nick Johnson Aug 13 '10 at 08:30
  • TypoonAE is no longer maintained (last release was in 2010). AppScale has since commercialized (http://appscale.com) the open source platform, is production ready, and has frequent releases. – Navraj Chohan Feb 17 '14 at 23:09
7

Use a high level framework that works on App-Engine. That way you can port your code to other servers when you want.

django has been patched and ported to work in the Appengine patch project and is the most used FW on appengine.

You may want to refer this step by step intro to running a django app on App engine

As far as the parallel infrastructure to run an app engine application is concerned, it is still way far. App Engine itself hasn't got as popular as people believed it to and google wanted it to be. Plus it is harder to develop on the builtin WebApp framework than on django.

Its quite unlikely to see a parallel infrastructure to run app engine application on, atleast for years to come. Rather it is likely to see django and other popular frameworks work out of the box on app engine and the work on this is currently underway in the referred project.

lprsd
  • 84,407
  • 47
  • 135
  • 168
  • 5
    The one problem I'll point out with app-engine-patch is that it doesn't use Django models; it instead exposes GAE models to the developer directly. That's not necessarily a bad thing; there's a few things you can do with Django models that just won't work with GAE, and an abstraction would be very leaky. But, this means that you'll be rewriting your models at a minimum when porting from GAE back to Django natively, and you'll be touching every query because of the syntax differences: not a trivial thing on a large project. – esm May 21 '09 at 13:39
  • I agree with esm, the basic stack is wsgiapp, cache, datastore and everything on top of that doesn't matter. Wsgiapp and the cache is a standard and persistent storage with bigtable characteristics are coming up fast (mongo, tokyo, memcachedb). – Koen Bok May 21 '09 at 14:32
  • 2
    "App Engine itself hasn't got as popular as people believed it to and google wanted it to be. Plus it is harder to develop on the builtin WebApp framework than on django." - I disagree with both assertions. And people are working on open-source infrastructure as we speak - see AppScale, for example. – Nick Johnson May 21 '09 at 20:55
  • You could use google-app-engine-django (http://code.google.com/p/google-app-engine-django/) instead, because that does give you a BaseModel class that emulates the standard Django model, so it should in theory be easier to port your application at a later stage. – Tom van Enckevort Oct 08 '09 at 11:04
3

So far, I found an experimental host called app-drop which is capable of hosting google app-engine projects. This should mean that it's atleast possible to run app engine projects outside of google's infrastructure.

This is however clearly not yet suitable for production.

Alterlife
  • 6,557
  • 7
  • 36
  • 49
  • I might be wrong, but my gut feeling would tell me not to count on this as a production alternative at this point. – Koen Bok May 21 '09 at 14:27
3

You can build AppEngine applications using the Django python framework (although the supported version is a bit behind the most recent Django release). Where you lose portability (at least right now) is when using GQL/BigTable for persistence. This is Google proprietary database platform. As Hank mentioned this is one of the biggest reasons to actually use AppEngine, but it is also the single largest locking point.

Here are a couple of links to Django support in AppEngine and GQL/BigTable:

Ken I.
  • 450
  • 1
  • 3
  • 8
1

The code should be mostly portable (they do a pretty good job of indicating which modules you can't use on AppEngine and which AppEngine-specific code corresponds to which forbidden modules), but the whole point of AppEngine is to get access to Google's infrastructure. There's not much point to writing your code to the AppEngine restrictions if you aren't going to be using their infrastructure.

Hank Gay
  • 70,339
  • 36
  • 160
  • 222
  • 1
    To clarify, the question really is: If I develop on google app-engine now, will I be able to migrate to another platform later, or is it a lock in? – Alterlife May 21 '09 at 11:25
  • In that case, becomingGuru probably has the answer for you. – Hank Gay May 21 '09 at 11:41
1

AppDrop is a proof of concept port of AppEngine to Amazon Web Services / Elastic Computing completed in April of 2008. It uses a flat file instead of BigTable and runs in a single instance, so there are scaling issues; but it's developer says it took him only four days, and perhaps these limitations can be addressed by others.

Thomas L Holaday
  • 13,614
  • 6
  • 40
  • 51
0

I did the reverse migration from vanilla Unix to app engine recently very easily by using WHIFF resources. Basically configure anything platform dependant as a resource and then swap/replace the resources on different configurations.

http://piopio.appspot.com/W1000_1000.resources

also see

http://aaron.oirt.rutgers.edu/myapp/docs/W1100_1200.wwiki

for a detailed example of resource swapping/configuration. (note: links may go away eventually, app is experimental.)

Aaron Watters
  • 2,784
  • 3
  • 23
  • 37
0

AppScale is the most mature open source implementation of Google App Engine. It's been in development since 2008 and currently support all four languages: Python, Java, Go, and PHP. It has users running their application in production today.

The FAQ explains what APIs are supported and what is lacking: https://github.com/AppScale/appscale/wiki/FAQs

(Disclaimer: I work on the project)

Navraj Chohan
  • 627
  • 4
  • 9
0

Check out typhoonae. it's in beta, but quite usable – we moved one of our apps to inhouse server running this stack.

Ivan Vovnenko
  • 121
  • 6