8

I was trying to run an A/B test using Firebase A/B testing, but just for specific versions of the app. There were no results and no users exposed to the experiment after 5 days of running it.

After that, I tried to run the same A/B test but without a version restrictions and the first results were available the next day.

So I thought that the way I was defining version restrictions was wrong. I tested it in Remote Config, but the Remote Config was returning the correct values when I applied the same version restriction. Here is how I defined the version restriction in both cases:

enter image description here

The app's version names are in the following format X.Y.Z (e.g. 6.51.0, 6.51.1)

Is there something that I am missing? Is the test significantly slower to start if specific versions are being targeted?

Vladimir Jovanović
  • 2,261
  • 2
  • 24
  • 43
  • Could you share your A/B test configuration? Are you using any additional targeting like activation event etc? – michalbrz Jun 12 '20 at 18:30
  • @michalbrz My A/B test configuration is pretty simple. The only targeting that I have is `applicationId` and version. There is no activation event. If I remove the version from targeting, the A/B test works, so it seems that it has to be something related to versions. But since the same version targeting works with remote config it seems to be some kind of bug in Firebase A/B testing. – Vladimir Jovanović Jun 15 '20 at 07:04
  • are you upload versions you want to test in firebase???? – Shahriyar Aghajani Jun 16 '20 at 11:12
  • @ShahriyarAghajani I am not sure that I understand the question. Some of the versions that I am targeting are uploaded to PlayStore, and some would be uploaded in the future, as I can't modify the A/B test when it starts. – Vladimir Jovanović Jun 16 '20 at 12:19
  • Versions are tricky in my experience with firebase/remote config, you could try the same with "regex" instead of "contains", in your case will be something like `6\.[5-6][5-6]` another tip, you could wipe your app's data from your test device to accelerate the collection of data to firebase console, so you don't have to wait one day. – yamilmedina Jun 17 '20 at 16:01
  • @yamilmedina I tested this on an app that has millions of users, so I don't think that data collection is a problem. And Remote Config works perfectly with versions. Only A/B tests have issues. I will try with regex, but more I play with this, more it seems that there is some bug in Firebase A/B Testing. – Vladimir Jovanović Jun 17 '20 at 17:05

1 Answers1

0

This should work. I have tried this many times. Make sure that you're targeting the correct value based on your app configuration. The Version targeting criterion...

  • For Android app, you should target the "versionName", not the "versionCode".
  • For iOS app, you should target the "CFBundleShortVersionString", not the "CFBundleVersion".

In addition, you should put the exact number on the console. For example, if the versionName or CFBundleShortVersionString is 1.0.0, then you should put 1.0.0 and not just 1.0.

looptheloop88
  • 3,026
  • 17
  • 20