21

I was going through the different types of release strategy and was confused between the Canary and A/B strategy. Both of them seems to be similar.

Everywhere I read on Canary is "Allows to test deployment by releasing the new version to a small group of them." and on A/B is "An A/B testing strategy targets a specific group of customers."

Then where the differences lie between them and what are the use cases of both?

References: https://azure.microsoft.com/en-in/overview/kubernetes-deployment-strategy/

Vikas Gupta
  • 10,779
  • 4
  • 35
  • 42

1 Answers1

40

A/B test's purpose is usually to see users' response (In a way, how much they like it) to a new UI, feature, etc. But you know that the new version works. So, you actually send randomly both versions of the application to all of them. It can be 50-50, 80-20, 90-10, anything. Sometimes the functionality is not even relevant. You might want to see which version attracts more clients and stuff like that.

Canary is more focused on how well works the new feature. Or if it actually works. It usually will be 90-10, 80-20, A >> B. Never 50-50, because if it goes wrong, you don't want half of your users to have a bad experience. So you are not positive if the new version is going to work as expected.

The most important difference (and this is what almost no one talks about) is that a canary testing has session affinity. So it doesn't send both versions to all users, but randomly sends some users to the new version, and keeps them on the same version.

suren
  • 7,817
  • 1
  • 30
  • 51
  • Correct me if I am wrong but won't A/B testing also be having session affinity? I know these are meant to serve two different purpose, canary testing can be a sort of test that occurs on production after having been promoted from developement and staging environments after internal testing, whereas A/B testing is just seeing how people may responsd to a feature by sending traffic to a subset of the the users. You want to release the canary deployment after getting the bug fixed but an A/B test could just be a test for how people react to change in the product they are using. – Anshuman Kumar Nov 24 '22 at 16:56
  • So if I were say applying an A/B test using something like istio, the method to implement would be the same only, configuration wise? – Anshuman Kumar Nov 24 '22 at 16:57
  • @AnshumanKumar A/B testing shouldn't have session affinity. Usually you would fix a bug in pre-production. that's why pre and pro should be the same, so you can re-produce these kind of things. Not sure if I understand Istio question. Istio can ne used for A/B testing with subsets, but session affinity is only based on http headers, because traffic comming from outside will always hit the ingress gateway. – suren Nov 25 '22 at 04:27
  • Istio does support cookie based and header based session affinity,but if at a really basic level, to implement a Canary release OR two versions of the application for an A/B test, we would basically need to specify the percentage of traffic being routed to each version (using a VirtualService in the case of Istio). But if we don't have session affinity in A/B testing, then a user could end up getting a different version of the application than they used the last time, right? – Anshuman Kumar Nov 25 '22 at 05:49
  • I kind of got the discussion sidetracked to Istio, but the gist of it is, if we don't have session affinity in A/B testing, a user can't receive the version of the application they had opted in for, right? For example, if I am XYZ social media company releasing a "brand new" Stories feature as initially as an opt in feature, and if users opt in, they should be able to get the Stories feature each time they re-open the application. I am not denying the other points as A/B testing and Canary are meant to serve different business use cases. – Anshuman Kumar Nov 25 '22 at 05:58
  • @AnshumanKumar Well, that's still canary, I would say, as it's a new feature that adds the stories. but you want the users to be able to say "I want to try". It would be considered A/B if you were going to dismiss one version and leave the other. In that case, you would want them to try both versions and decide. – suren Nov 26 '22 at 06:30