How to correctly use Firebase-Analytcs for A/B testing?

Question

Background

Google has published a whole new API and services for A/B testing, using Firebase Analytics and Firebase Remote Config.

The problem

While I do try out the service for other purposes (here and here), I also need to use it for A/B testing.

Thing is, I can't find out how to make it show the needed information for analyzing the results of variants of the experiments.

It just doesn't show (on this website), or I don't look at the right place:

What I've tried

In order to try a single experiment, I did as the tutorial say I should do, including :

1. Initialization in code is about the same as in the samples :

   boolean isDebug = AppComponentsHelper.isInDebugFlavour(context);
   mFirebaseRemoteConfig = FirebaseRemoteConfig.getInstance();
   FirebaseRemoteConfigSettings configSettings = new FirebaseRemoteConfigSettings.Builder().setDeveloperModeEnabled(isDebug).build();
   mFirebaseRemoteConfig.setConfigSettings(configSettings);
   final HashMap<String, Object> defaults = new HashMap<>();
   for (ExperimentType experimentType : ExperimentType.values())
       defaults.put(experimentType.experimentId, experimentType.defaultValues);
   mFirebaseRemoteConfig.setDefaults(defaults);
   
   long cacheExpiration = isDebug ? 0 : TimeUnit.HOURS.toSeconds(1);
   mFirebaseRemoteConfig.fetch(cacheExpiration)
           .addOnCompleteListener(new OnCompleteListener<Void>() {
               @Override
               public void onComplete(@NonNull Task<Void> task) {
                   if (task.isSuccessful()) {
                       mFirebaseRemoteConfig.activateFetched();
                       final FirebaseAnalytics firebaseAnalytics = FirebaseAnalytics.getInstance(context);
                       for (ExperimentType experimentType : ExperimentType.values()) {
                           String experimentVariant = FirebaseRemoteConfig.getInstance().getString(experimentType.experimentId);
                           firebaseAnalytics.setUserProperty(experimentType.userProperty, experimentVariant);
                       }
                   }
               }
           });
   

   This is the enum of ExperimentType:

   public enum ExperimentType {
     PURCHASES_OFFERED_PRODUCTS("purchases_offered_products_1", "Experiment1", new Gson().toJson(new String[]{"something", "something2"}));
   
     public final String experimentId,defaultValues,userProperty;
   
     ExperimentType(final String experimentId, final String userProperty, final String defaultValues) {
         this.experimentId = experimentId;
         this.defaultValues = defaultValues;
         this.userProperty = userProperty;
     }
   

   }

2. Creating a new user property for each experiment (just one for now).

3. In remote config screen, add 2 variants (default and conditional).

4. Upon loading the remote config , calling setUserProperty (see #0), as shown in the tutorial, to point to the current variant's parameter value.

5. In order to track an event, I called:

   firebaseAnalytics.logEvent(eventName, new Bundle())
   

   One of the calls (only one for now), as you might have guessed is when eventName is "subscription_premium_purchase" .

6. When running the app, I caused it to use both variants (one for each clean installation, of course).

But as I've shown, I can't find a way to determine which variant caused the event to be called more than the other, for example.

The questions

1. What might be missing here? What have I done wrong? Why can't I see any variant that I've used?

2. What is the user property for, anyway? Is it used as a "glue" between Remote-Config and Analytics? What am I supposed to enter in the textbox there (it says "Press 'enter' to apply") ? Doesn't it suppose to show the various variants? Other filters items have "types", yet this one doesn't. There is nothing to choose from the user property. Even when creating a user property, I can't see any settings for it (except for description). I can't even delete it. There is no way to see the conditions I've created in the graph.

3. Do I need to use BigQuery, perhaps?

4. I've noticed that the "fetch" function sometimes doesn't get the listener to be called at all. How could it be? Can I add a timeout for this?

Answer 1

I think the key confusion here is related to the fact that you don't see your values as auto-complete suggestions when you chose the Experiment user property. This is a feature that is being considered. But if you just type in your Experiment values, it will apply that as a filter on your reporting so you can see how many "subscription_premium_purchase" events were logged by users in one variant versus the other.