How can you choose variations for several content experiments?

Question

I'm using the Javascript API for Google Analytics Content Experiments and it seems like you have two choices:

1. Run one experiment at a time and have Google assign the variation, which you can get using chooseVariation(), or
2. Run several experiments at a time and assign the variation yourself using setChosenVariation()

(1) lets Google control the number of users assigned to each variation, which I need, but I have several experiments. This seems pretty basic. What am I missing?

Answer 1

I don't think you're missing anything. My understanding is that the GA Experiments API allows you to allocate a visitor to any version of a single A/B/N test with the Choose Variation command :

cxApi.chooseVariation();

However, this function is only available if you specify the experiment ID when loading the api.js on the page.

If what you want to do is to run several A/B tests on one page at the same time, this is really multi-variate testing and unfortunately this is not available 'out-of-the-box' in GA Content Experiments (it used to be available with Google Website Optimizer). You can still 'fake' it using your own bucketing, with code like :

<!-- 1. Load the Content Experiments JavaScript Client -->
<script src="//www.google-analytics.com/cx/api.js"></script>

<!-- 2. See if the visitor has seen an experiment already -->
<script>
// loop through all active experiments server-side. 
cxApi.getChosenVariation(
 $experimentId
 );

<!-- 3. Bucket the visitor if he isn't bucketed already -->
// loop through your live experiments 
  cxApi.setChosenVariation(
    $chosenVariation,             // The index of the variation shown to the visitor
    $experimentId                 // The id of the experiment the user has been exposed to
  );
</script>

You will need to decide whether you want a visitor to only view one experiment at a time or enter multiple experiments concurrently.

The first option is probably more reliable but means you will have to split your visitor in quite a lot of buckets, meaning your tests will take a long time before returning a result.

If you use the second option, you need to be careful : if your experiments are not independent, you won't be able to trust the test results from Google Analytics, as the statistical tests used in the Content Experiments assume independence. Even if your experiments are independent, you'll need to have an equal split among all variation combinations.

I hope that helps !

Answer 2

when running many simultaneously experiments the problem is that if you have visitors from one experiment that interact with elements of another experiment running on your site then it is difficult to account for these interactive effects. This will skew your data and lead to some faulty conclusions.if you’re absolutely confident in how to analyze multiple concurrent experiments, it’s okay be safe and run one experiment at a time.

Answer 3

I solved this using a node.js script.

Out-of-the-box Google Content Experiments makes you choose between:

• Load cxApi patched only for 1 experiment
• Load cxApi plain, and lose the ability to call chooseVariation() function, making you implement your own choosing algorithm (which is a shame, as google's multi-armed bandit idea is really smart).

So I did some reverse engineering on how google cxApi js is patched for 1 experiment, it turns it out it just bundles the experiment info (the variations weights) in the middle of the file, I experimented touching the bundle manually adding multiple experiments, and now I can call cxApi.chooseVariation(experimentId) for each experiment, and works nicely!

Weights change every 12hs, so I automated it, I created a node.js little app that will download cxApi from google, and then repeatedly download it for each experiment you specify, and patch a cxApi with all your experiments on it.

Voila!, multiple experiments on any page.

Here's the repo: https://github.com/benjamine/mutagen, fork at will

this node.js app will patch cxApi for multiple experiments, and will also pack (and minify) your variations

Answer 4

This solution works very well for me. The benefits are:

• no node.js required
• no changes to site needed - site remains fully cacheable
• Google Multi-Armed-Bandit approach used, no patch
• easily maintainable

Javascript Module providing all functionality for executing multiple Google Experiments:

var runExperiment = (function(jQ) {

    var apiUrl = '/ga-experiments-api.php?preview=true&experimentId=',
        experimentId,
        variations;

    var chooseAndApplyNewVariation = function() {
        if (typeof jQ !== 'undefined') {
          jQ.ajax({
              type: 'get',
              url: apiUrl + experimentId,
              success: function(data, status){
                  var choosenVariation = data.choosenVariation;

                  if (typeof choosenVariation !== 'undefined' && choosenVariation >= 0) {
                      cxApi.setChosenVariation(choosenVariation, experimentId);
                      applyVariation(choosenVariation);
                  }
              }
          });
        }  
    };

    var applyVariation = function(chosenVariation) {
        var variationFunction = (typeof variations[chosenVariation] === 'function') ? variations[chosenVariation] : false;

        if (variationFunction) {

            variationFunction.apply();
            sentGaEvent();
            console.log(experimentId, chosenVariation);
        }
    };

    var sentGaEvent = function() {
        if (typeof ga !== 'undefined') {
            ga('send', 'event', 'experiment', 'view');
        }
    };

    return function(experiment) {
        experimentId = experiment.id;
        variations = experiment.variations;

        if (typeof cxApi !== 'undefined') {

            var chosenVariation = cxApi.getChosenVariation(experimentId);

            if (chosenVariation >= 0) {

                applyVariation(chosenVariation);

            } else {

                chooseAndApplyNewVariation();
            }
        }
    };
})(jQuery);

Javascript Snippet for running single experiment - can be integrated multiple times on page:

(function(jQ) {
    var experiment = {
        'id': 'RHwa-te2T_WnsuZ_L_VQBw',
        'variations': [
            function() {},
            function() {
                jQ('#nav #menu-item-2000927 a').text('Shop + Abo');
            }]
    };

    runExperiment(experiment);
}(jQuery));

PHP (API) for generating new variations through Google's API, I used this class:

https://github.com/thomasbachem/php-gacx

<?php

use UnitedPrototype\GoogleAnalytics;

require_once dirname(__FILE__) . '/libs/googleAnalytics/Experiment.php';

$experimentId = (!empty($_GET['experimentId'])) ? $_GET['experimentId'] : null;

$returnData = array(
    'experimentId' => $experimentId,
);

try {
    $experiment = new GoogleAnalytics\Experiment($experimentId);

    $variation = $experiment->chooseNewVariation();

    if (is_integer($variation)) {

        $returnData['success'] = true;
        $returnData['choosenVariation'] = $variation;
    }

} catch (Exception $exception) {

    $returnData['success'] = false;
    $returnData['error'] = $exception;
}

header('Content-Type: application/json');

echo json_encode($returnData);