A controlled comparison of the effectiveness of variants of a website, email, or other commercial product. Usually, a novel or treatment condition is compared against a baseline or control condition to determine the effect of the treatment condition.
A/B test, or split test, is a business term for an experiment in which users are randomly exposed to one of several variants of a product, often a website feature.
The Response or Dependent Variable is most often count data (such as clicks on links or sales) but may be a continuous measure (like time on site). Count data is sometimes transformed to rates for analysis.
Because they create temporary variants of 'live' websites, online A/B tests must overcome several challenges not common in traditional experiments of human preference. For example, differential caching of test versions may degrade website performance for some versions. Users may be shown multiple variants if they return to a website and are not successfully identified with cookies or by login information. Moreover, nonhuman activity (search engine crawlers, email harvesters, and botnets) may be mistaken for human users.
Useful References:
