A/B testing on App Engine?

Question

What would be the simplest implementation of an A/B testing system running on App engine?

I'm especially keen towards performance implications of using Datastore for back-end (with looong query times), and database design.

The more I read your question the less I understand. How's A/B related to datastore? o_O Could you please show us an example of a test you'd do? — Vladimir Dyuzhev, Apr 15 '10 at 00:15

score 5 · Answer 1 · answered Sep 15 '11 at 06:39

5

Have a look to Gae/Bingo, it's an A/B split-testing framework for App Engine inspired by A/Bingo.

More information here.

answered Sep 15 '11 at 06:39

systempuntoout

71,966
47
171
241

1

Author of GAE/Bingo here --> we spent a particularly long time making sure that data is persisted in the backend without impacting pageload times (persisting data via task queues in the background, etc). Check out systempuntoout's links for more. – kamens Oct 01 '11 at 23:41

score 4 · Answer 2 · answered Dec 11 '10 at 17:44

You could deploy two versions of your application:

appcfg.py update -V "A" mysiteA/
appcfg.py update -V "B" mysiteB/

And then create a third version that simply chooses whether to proxy a user to A.latest.mysite.appspot.com or B.latest.mysite.appspot.com.

score 2 · Answer 3 · answered Feb 29 '12 at 18:58

2

It is now generally available in SDK 1.6.3 as Traffic Splitting feature: http://code.google.com/appengine/docs/adminconsole/trafficsplitting.html

answered Feb 29 '12 at 18:58

alex

2,450
16
22

score 1 · Answer 4 · answered Apr 14 '10 at 09:32

1

Assuming you want to test different versions of your app, I would suggest using a simple bit of WSGI middleware. Build something that directs x% of users to one WSGI app, and the remainder to another, sharded by whatever suits - user ID, IP address, etcetera. This should be pretty straightforward to implement, and you can pile whatever you like on top of it.

answered Apr 14 '10 at 09:32

Nick Johnson

100,655
16
128
198

Doable, but kind of overly complicated. Read how easy A/B testing should be: http://www.bingocardcreator.com/abingo – Vladimir Dyuzhev Apr 14 '10 at 13:43
Something like that is certainly possible in Python as well as ruby. I was going for a more general (though higher overhead) approach. :) – Nick Johnson Apr 15 '10 at 09:43

score -1 · Answer 5 · answered Apr 14 '10 at 13:40

A/B test requires to show page A to some users, while page B to some other users.

App Engine has nothing to do with it. App Engine is a way to deploy applications, not direct user along the pages.

It's the function of the web framework you use to serve one page or another based on user cookie/session.

In a simple way it could be done like this:

Get user cookie
Find it in datastore
Found? Use the same set of pages (A or B) as the last time
Not found? Choose A or B randomly, save the choice into datastore along with cookie
(May be) Place the choice into session for fast access

Then, in specific controllers/views, based on selected A or B, serve/redirect user to page A or page B. Record the outcome (whatever your outcome is -- sale, registration, ...) into datastore.

That can be done for any web framework. You didn't even told which one you use ;)

A/B testing on App Engine?

5 Answers5