How to do simple AB testing in iOS

Question

I am looking to split up my user base to 10 group and show 10 different UI and see how they feel about it.

so each user group will have single type of UI always.

i.e Let's say I have 10k users and when I roll out my next release when user install I will be showing for 1000 user 1 UI and for another 1000 user 1 UI like all 10K users.

I know this can be done with the help of AB testing framework.

Basically I want to call one API at the launch of app and it has to return value between 1 to 10 then I can store it in my keychain and next time when app is launched I will see if it's already there in keychain and I will not call the API.

So basically the API will know how many requests has come and it'll divide and send right values back

so based on value in keychain I will show different , different UI and here AB testing framework's job would be giving me value 1 to 10 the API part.

There are so many AB testing framework available online.But I couldn't find any framework that suits my needs.

any help is appreciated !

Answer 1

The best approach would be splitting the users into groups in data base and let login API or some other API return some flag to indicate what group each user belongs to and you can show UI accordingly.

But if that's not possible

Then the simplest approach would be generating a random number between 1-10 and keeping it in keychain and showing a particular UI for it so that next time when you launch the app you can look out for the value in Keychain and if its not there then you can create a new random value and store it in the keychain.This way you will show the same UI for that user always.

This splitting approach is not 100% accurate but its close enough I would say

arc4random_uniform

- (NSInteger)randomNumberBetween:(NSInteger)min maxNumber:(NSInteger)max
{
    return min + arc4random_uniform((uint32_t)(max - min + 1));
}

if you take sample of these random numbers 10000 times, you can see each number coming 900-1000 times which is 9-10% and its close enough

 for(int i=0;i<10000;i++){
      NSLog(@"random:%ld",[self randomNumberBetween:1 maxNumber:10]);
    }

Seconds of Current time

you can take the seconds of current date and time and if the second is between 1-6 then you can save value 1 in keychain and for 7-12 you can save value 2 in keychain etc..54-60 you can keep value 10 in keychain.

Others

you can consider splitting the users based on Geography or country or timezone and doing this also has its own pit falls.

Like this you can devise your own strategy to split the user

but if none of the suggestions above fits your criteria then the best approach would be to look for third party AB testing frameworks but If it's going to be implemented in enterprise scale they might charge some money for it.

If I come across any such framework that provides this particular functionality alone as you asked I would update it here.

Answer 2

I would like to attribute the credit of this answer to this post as he has pointed out FireBase Remote Config and A/B testing.

As questioner asked I will explain the steps involved in that to achieve it.

Configuration on server

• Visit https://console.firebase.google.com/ and sign in with your google account.

• Choose Create project and Click iOS

• Key in app id and nick name and click register app

• It'll show a link to GoogleService-Info.plist download then drag & drop it in the project

• Choose Next

• It'll show you Run your app to verify installation you can choose skip this step

• Choose remote config from the landing page

• Choose Add variable and enter a variable name of your choice but I enter ABTestVariationType and leave value empty and choose Publish changes

• Choose A/B testing from the side bar then click Create Experiment then Choose Remote config

• In the upcoming pop up Enter the name of your choice I enter as A/B test POC enter some description about it and that's optional anyway

• In the the target users choose your app id and in the percentage of target users choose 100% and click Next then it'll show the variants section

• In the variants section there will be a general category named Control group with 50% loaded by default and a variant box with 50% filled in and empty box and you can enter any name in that but I would enter variant 2.Now click add a parameter 8 times now you can see each variant has 10% and name all the variants and I would name variant 3,variant 4 to variant 10.

• In the same variants section click Add Parameter from Remote config
  Now you can see the a box appearing besides each variation parameter.You can enter unique value to identify each flavour.I would enter value 1 for the first variant and 2 for the second variant like that I will finish up with value 10 for the last variant and click Next

• Then goal section appears you can choose one of it but I would choose Retention(15+) days and click Review and click start experiment and in prompt that's appearing choose start again

Integrating in the app

1. Add the following pods in your project

   pod 'Firebase/Core'
   
   pod 'Firebase/RemoteConfig'
   

2. Drag and drop the GoogleService-Info.plist that was downloaded during the server configuration

3. Initiate the firebase with following boiler-plate code

     @import Firebase;
   
     -(BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions(NSDictionary *)launchOptions 
    {
   
      [FIRApp configure];
   
      return YES;
   
    }
   

4.Have the class RcValues which is another boiler-plate code in your project

     #import "RcValues.h"  

     @import Firebase;

     @implementation RcValues

     +(RcValues *)sharedInstance 
     {

      static RcValues *sharedInstance = nil;

      static dispatch_once_t onceToken;

      dispatch_once(&onceToken, ^{

        sharedInstance = [[RcValues alloc] init];
      });

       return sharedInstance; 

     }

   -(id)init{

     self=[super init];

     if(self)
     {

      [self AcivateDebugMode];

      [self LoadDefaultValues];

      [self FetchCloudValues];

     }

     return self;

    }

   -(void)LoadDefaultValues    
   { 

    [FIRRemoteConfig.remoteConfigsetDefaults:
    @{@"appPrimaryColor":@"#FBB03B"}]; 

   }

   -(void)FetchCloudValues
   {

     NSTimeInterval fetchInterval=0;

    [FIRRemoteConfig.remoteConfigfetchWithExpirationDuration:
    fetchInterval completionHandler:^(FIRRemoteConfigFetchStatus 
    status, NSError *_Nullable error) 

     {

       NSLog(@"error:%@",error);

      [FIRRemoteConfig.remoteConfig activateFetched];

      }]; 
    }

  -(void)AcivateDebugMode{ //    

   FIRRemoteConfig.remoteConfig.configSettings=debugSettings;

   FIRRemoteConfigSettings *config = [[FIRRemoteConfigSettings alloc] initWithDeveloperModeEnabled:YES];

   FIRRemoteConfig.remoteConfig.configSettings=config;
  }

  @end

5.Invoke the class in appdelegate didFinishinglaunchoptions

    RcValues *Obj=[RcValues sharedInstance]; 

This will download the keyvalue for ABtesting

6.Use the below code to get the AB testing key from firebase to your app

    self.flavourNumber.text=[FIRRemoteConfig.remoteConfig
    configValueForKey:@"ABTestVariationType"].stringValue;

Based on the key value you can show different UI as you wish.

Firebase will take care of sending the right value and you don't have to worry about divide the users into groups by yourself.

P.S

Please follow the below tutorials for more detailed info this is just a summary and I will try to summarise or add more pictures when I have free time to make it for easier understand if possible I will try to add sample project in github and link it here

firebase-tutorial-ios-ab-testing

firebase-remote-config-tutorial-for-ios

Answer 3

Imagine changing fonts, colour or some values in your iOS app without submitting a new build. It's pretty easy using Remote config. This tutorial will teach you A/B testing, but before A/B testing I would recommend you to look around Remote Config.