1

I am all new to Kubernetes. That said: I'm looking to add failover capabilities to an existing dockerized service which you'd probably consider as legacy service, in the sense that it requires an active/standby failover handling (full rationale will be rather long to describe, it has to do with spatial indexing and geo data processing where for a given scenario having multiple active instances will make things worse in regards to scalability and performance, e.g. sharding would get incredibly hard to handle).

My service is an HTTP REST service (in C++) and uses is a DB client to a distributed RQLite DB which stores all data managed by this service. I want to achieve the following:

  • I want to have 3 RQLite services up and running continuously at all times, distributed to three nodes if possible (NB: RQLite itself uses RAFT with a default group of three).
  • I also want to have 2-3 services of my own app running all time, one of them must be the master all others must remain in standby.
  • Services must be able to know another (e.g. resolve IPs of RQLite instances)

I could have missed it, but don't whether active/standby is actively supported by Kubernetes. I've found some dated answers like the one here (How should application using active/passive redundant model be containerized using kubernetes?).

But before looking into many details which are all new to me, I'd like to get your rationale whether Kubernetes is actually the correct foundation for such kind of scenario (Nomad might be better for such "legacy" apps?)?

It would be great if you could provide a minimal example to demonstrate how Kubernetes can solve my failover wows.

Rico
  • 58,485
  • 12
  • 111
  • 141
benjist
  • 2,740
  • 3
  • 31
  • 58
  • Why do you want to have one of your services be a master and the others standby idle? I think this actually makes it harder to implement than just running 2-3 replicas with all of them getting traffic. – Brian Pursley Jul 09 '20 at 01:13
  • @BrianPursley The service is for example handling geofencing events, e.g. a location update within a fence triggers a fence entry event. Services would have to make strong guarantees that they are in sync for any update. Example: A location update received by instance 1 triggers a fence entry event. Another location update received by instance 2 must guarantee to not trigger a duplicate entry event. To make this scalable and avoid synchronization, it needs to be optimized based on spatial attributes. – benjist Jul 09 '20 at 12:01

1 Answers1

2

Sure, regarding Kubernetes. You can also use Nomad, I don't think there is no right or wrong so it's mostly about opinion and what you would like to try.

I think if you just want to use a single Kubernetes Service, you can just make use of ReadinessProbes in your replicas. You can use a Deployment to manage your replicas. This is assuming your application has a leader election mechanism (like RAFT) that only has that 1 pod/instance receiving traffic at one time. The leader will always pass the readiness probe while the non-leader will fail the readiness probe. A Kubernetes service will not forward traffic to pods where their readiness probe is not passing.

Another option is to set up your own proxy like nginx/haproxy in front of 2 or more services, that proxy will do the health checks and will forward only to the 'active' services.

Similar options are available with Nomad.

Rico
  • 58,485
  • 12
  • 111
  • 141