Which Kubernetes pattern is appropriate for the peer to peer scenario where peers have slightly different configuration?

Question

I'm trying to run private stellar blockchain infrastructure on kubernetes (not to join to existing public or test stellar network) but my question can be generalized to the scenario of running any peer to peer services on kubernetes. Therefore, I will try to explain my problem in a generalized way (hoping that it can yield answers that are applicable to any similar topology running on the kubernetes).

Here is the scenario:

I want to run 3 peers (in kube terms: pods) which are able to communicate with each other in a decentralized way but the problem lies in the fact that each of these peers has a slightly different configuration. In general, configuration looks like this (this is an example for pod0):

NETWORK_PASSPHRASE="my private network"

NODE_SEED=<pod0_private_key>

KNOWN_PEERS=[
    "stellar-0",
    "stellar-1",
    "stellar-2"]

[QUORUM_SET]
VALIDATORS=[ <pod1_pub_key>, <pod2_pub_key> ]

The problem lies in the fact that each pod would have different:

NODE_SEED
VALIDATORS list

My first idea (before realizing this problem) was to:

Create config map for this configuration
Create statefulset (3 replicas) with headless service to enable stable reachability between pods (stellar-0, stellar-1, stellar-2...etc.)

Another idea (after realizing this problem) would be to:

Create separate config maps for each peer
Create statefulset (1 replica) with service

I'm wondering if there is any better solution/pattern that could be utilized for this purpose rather than running completely same services with slightly different configuration as separate entities (statefulset, deployment..) with their separate service through which these peers would be available (but this kind of defeats a purpose of using kubernetes high level resources which enable replication)?

Thanks

score 6 · Accepted Answer · answered Sep 24 '18 at 04:09

So you can have a single ConfigMap with multiple keys each one uniquely meant for one of your replicas. You can also deploy your pods using a StatefulSet with an initContainer to setup the configs. This is just an example (You'll have to tweak it to your needs):

ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: stellar
  labels:
    app: stellar
data:
  stellar0.cnf: |
    NETWORK_PASSPHRASE="my private network"    
    NODE_SEED=<stellar0_private_key>    
    KNOWN_PEERS=[
        "stellar-0",
        "stellar-1",
        "stellar-2"]    
    [QUORUM_SET]
    VALIDATORS=[ <stellar1_pub_key>, <stellar2_pub_key> ]

  stellar1.cnf: |

    NETWORK_PASSPHRASE="my private network"
    NODE_SEED=<stellar1_private_key>
    KNOWN_PEERS=[
        "stellar-0",
        "stellar-1",
        "stellar-2"]

    [QUORUM_SET]
    VALIDATORS=[ <stellar0_pub_key>, <stellar2_pub_key> ]

  stellar2.cnf: |

    NETWORK_PASSPHRASE="my private network"
    NODE_SEED=<stellar2_private_key>
    KNOWN_PEERS=[
        "stellar-0",
        "stellar-1",
        "stellar-2"]

    [QUORUM_SET]
    VALIDATORS=[ <stellar0_pub_key>, <stellar1_pub_key> ]

StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: stellarblockchain
spec:
  selector:
    matchLabels:
      app: stellar
  serviceName: stellar
  replicas: 3
  template:
    metadata:
      labels:
        app: stellar
    spec:
      initContainers:
      - name: init-stellar
        image: stellar-image:version
        command:
        - bash
        - "-c"
        - |
          set -ex
          # Generate config from pod ordinal index.
          [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
          ordinal=${BASH_REMATCH[1]}
          # Copy appropriate conf.d files from config-map to emptyDir.
          if [[ $ordinal -eq 0 ]]; then
            cp /mnt/config-map/stellar0.cnf /mnt/conf.d/
          elif [[ $ordinal -eq 1 ]]; then
            cp /mnt/config-map/stellar1.cnf /mnt/conf.d/
          else
            cp /mnt/config-map/stellar2.cnf /mnt/conf.d/
          fi
        volumeMounts:
        - name: conf
          mountPath: /mnt/conf.d
        - name: config-map
          mountPath: /mnt/config-map

      containers:
      - name: stellar
        image: stellar-image:version
        ports:
        - name: stellar
          containerPort: <whatever port you need here>
        volumeMounts:
        - name: conf
          mountPath: /etc/stellar/conf.d <== wherever your config for stellar needs to be

     volumes:
     - name: conf
       emptyDir: {}
     - name: config-map
       configMap:
         name: stellar

Service (if you need to expose it)

apiVersion: v1
kind: Service
metadata:
  name: stellar
  labels:
    app: stellar
spec:
  ports:
  - name: stellar
    port: <stellar-port>
  clusterIP: None
  selector:
    app: stellar

Hope it helps!

score 1 · Answer 2 · answered Dec 27 '19 at 21:17

It is worth stating: Kube's main strength is managing scalable workloads of identical Pods. That's why the ReplicaSet exists in the Kube API.

Blockchain validator nodes are not identical Pods. They are not anonymous ; they are identified by their public addresses which require unique private keys.

Blockchain nodes which serve as RPC nodes are simpler in this sense; they can be replicated and RPC requests can be round robined between the nodes.

There is value to using Kube for blockchain networks; but if deploying validators (and boot nodes) feels like it goes against the grain, that's because it doesn't neatly fit into the ReplicaSet model.

Which Kubernetes pattern is appropriate for the peer to peer scenario where peers have slightly different configuration?

2 Answers2