5

We have a problem with the disproportion of AB test variants. In theory, variants should be almost equally exposed for both variants, which actually worked quite good with previous AB tests that we made. ab test config 1 ab test config 2

As you can see on the screenshots with current total users count 161k, variants have 75.4k and 85.9k. We have run this test a couple of times with slight code changes, but every time the disproportion was present similarly.

In the running AB test, we set as a default value for remote config parameter "ios_show_recent_searches_v2" empty string.

The code that is running the ab test with activation event looks like this:

@objc class FireBaseService: NSObject {
    struct Constants {
        static let recentSearchesABTestToggleKey: String = "ios_show_recent_searches_v2"
        static let abTestNotAvailableKey: String = "N/A"
    }

    @objc class func start() {
        FirebaseApp.configure()
        let remoteConfig = RemoteConfig.remoteConfig()

        let oldShowRecentSearches = showRecentSearches

        remoteConfig.fetch(withExpirationDuration: 0.0) { (status, error) -> Void in
            DispatchQueue.main.async {
                if status == .success {
                    remoteConfig.activateFetched()
                    FireBaseService.notifyRecentSearchesVisiblityChangeIfNeeded(oldValue: oldShowRecentSearches)
                    FireBaseService.sendRecentSearchesActivationEventIfNeeded()
                }
            }
        }
    }  

    private class func hasConfigs(for key: String) -> Bool {
        return (RemoteConfig.remoteConfig()[key].stringValue?.count ?? 0) > 0
    }
}

// MARK: - Recent Searches A/B Test
extension FireBaseService {

    @objc class var showRecentSearchesValue: String {
        if hasConfigs(for: Constants.recentSearchesABTestToggleKey) {
            return RemoteConfig.remoteConfig()[Constants.recentSearchesABTestToggleKey].stringValue ?? Constants.abTestNotAvailableKey
        }
        return Constants.abTestNotAvailableKey
    }

    class var showRecentSearches: Bool {
        return RemoteConfig.remoteConfig()[Constants.recentSearchesABTestToggleKey].boolValue
    }

    private class func sendRecentSearchesActivationEventIfNeeded() {
        if FireBaseService.showRecentSearchesValue != Constants.abTestNotAvailableKey {
            AnalyticsServiceAdapter.activateRecentSearchesExperiment()
        }
    }

    private class func notifyRecentSearchesVisiblityChangeIfNeeded(oldValue: Bool) {
        if oldValue != showRecentSearches {
            NotificationCenter.default.post(name: Notification.Name(FireBaseService.ShowRecentSearchesNotification), object: nil)
        }
    }
}

The function GEFireBaseService.start is invoked on the application launch.

sliwinski.lukas
  • 1,412
  • 1
  • 18
  • 28

0 Answers0