7

I would like to setup an inifinispan cluster in a kubernetes environment.

DNS_PING is the discovery protocol. It works with pod deployed with DeploymentConfig settings. However, when we switch the settings from DeploymentConfig to StatefulSet, it does not work.

It is because we cannot query the IP of a pod directly with its name (pod name), but with (pod name).(headless service created) pattern.

jgroups is new to me. Appreciate if anyone may suggest us how to configure the DNS_PING properly for this scenario. Many thanks.

William Wong
  • 319
  • 3
  • 12
  • Hmmm, don't think I have an example ready for that. I recently wrote (infinispan-kubernetes)[https://github.com/infinispan-demos/infinispan-kubernetes] example but I think it uses `KUBE_PING` which requires cluster role binding. It should be possible to switch that example to DNS_PING but I've not go around to doing it yet... – Galder Zamarreño Dec 14 '18 at 09:19
  • I might be able to give this a go in the next few days. Can you tell which Kubernetes distribution are you using? Is it OpenShift? Minikube? Or some other? – Galder Zamarreño Dec 14 '18 at 10:23
  • Hi @GalderZamarreño, Yes, I tried that with OpenShift. – William Wong Dec 17 '18 at 01:47
  • Sorry didn't have time for this. The templates [here](https://github.com/jboss-container-images/jboss-datagrid-7-openshift-image/blob/datagrid73-dev/services/cache-service-template.yaml) are used for the dedicate OpenShift services. They use stateful sets and DNS ping, you might find there what you want. – Galder Zamarreño Jan 14 '19 at 15:30
  • Hi @GalderZamarreño, Thanks for the help. The cluster can be up and running with stateful set. However, when one of the node gone, we hit error message like this. It seems that the FD tries to recover the connection by connecting to the host directly with the pod name (oauth2-1) 03:43:37,097 WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,oauth2-2) JGRP000032: oauth2-2: no physical address for oauth2-1, dropping message – William Wong Mar 22 '19 at 03:46
  • That's just a warning message :| – Galder Zamarreño Apr 09 '19 at 08:49
  • These guys got a StatefulSet working via DNS_PING https://github.com/codecentric/helm-charts/tree/master/charts/keycloak – Max Lobur Dec 16 '19 at 16:05

1 Answers1

0

I am working on something similar that I hope it helps:

When you start a pod on statefulset you can retrieve the DNS local with an internal shell and showing the host file, like the following:

% kubectl exec custom-infinispan-0 -it -- sh
sh-4.4$ cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.xxx.xx.xx    stafulset-0.<your-service>.default.svc.cluster.local    statefulset-0

In case of infinispan you need to load the proper query as -D parameter using JAVA_OPTS.

Something like the following can be used:

apiVersion: v1
kind: ConfigMap
metadata:
  name: <your-thingie>-infinispan
data:
  INFINISPAN_CACHE_STATISTICS: "true"
  USER: "admin"
  DB_ADDR: "mariadb"
  DB_PORT: "3306"
  DB_USER: "<>"
  DB_DATABASE: "<>"
  JAVA_OPTIONS: "-Dinfinispan.cluster.stack=k8s -Djgroups.dns.query=<your-thingie>-infinispan.default.svc.cluster.local"

The big deal is that you will need to generate your custom container, because infinispan does not allways include the configurations for the cluster stack. You can find them at https://github.com/infinispan/infinispan/tree/main/core/src/main/resources/default-configs

If you need more info about your resolution you can also execute the following:

% kubectl describe svc <your-service-name>
Alberto Soto
  • 111
  • 3