Kubernetes: Haproxy ingress isn't taking on any addresses

Question

I'm running Kubernetes v1.25.9+rke2r1. I have a Metallb correctly setup with an external ip address and haproxy ingress controller. I need to expose a cockroachdb instance via ingress and I set up that config in the crdb instance correctly and the ingress spins up correctly. but without any ip address. And therefore when I click the page goes into timeout, if I telnet the port it goes into timeout aswell.

k get ingress -A
NAMESPACE              NAME              CLASS     HOSTS                                                     ADDRESS   PORTS   AGE
cockroachdb-instance   sql-cockroachdb   haproxy   cockroachdb.dev-service-stage.k8s.mi1.prod.cloudfire.it             80      15m

From the haproxy pod logs I don't see any errors, to my knowledge the ingress and the corresponding service are setup correctly. What can I do in order to further debug this or fix this?

k describe ingress sql-cockroachdb -n cockroachdb-instance
Name:             sql-cockroachdb
Labels:           app.kubernetes.io/component=database
                  app.kubernetes.io/instance=cockroachdb
                  app.kubernetes.io/managed-by=cockroach-operator
                  app.kubernetes.io/name=cockroachdb
                  app.kubernetes.io/part-of=cockroachdb
                  app.kubernetes.io/version=v23.1.4
                  crdb=test
Namespace:        cockroachdb-instance
Address:          
Ingress Class:    haproxy
Default backend:  <default>
Rules:
  Host                                                     Path  Backends
  ----                                                     ----  --------
  cockroachdb.dev-service-stage.k8s.mi1.prod.cloudfire.it  
                                                              cockroachdb-public:sql (10.222.19.88:26257,10.222.196.89:26257,10.222.30.93:26257)
Annotations:                                               crdb.io/last-applied:
                                                             UEsDBBQACAAIAAAAAAAAAAAAAAAAAAAAAAAIAAAAb3JpZ2luYWyMk0GT0zAMhf+LzrG3pbvQ5ki5cIEZYLgwHGRbaTxxbK+sFDqd/HfGbelsoYe9aZSn7+kpyRFGEnQoCO0RAhoKpV...
                                                           field.cattle.io/publicEndpoints:
                                                             [{"addresses":[""],"port":80,"protocol":"HTTP","serviceName":"cockroachdb-instance:cockroachdb-public","ingressName":"cockroachdb-instance...
Events:
  Type    Reason  Age                From                Message
  ----    ------  ----               ----                -------
  Normal  CREATE  28m                ingress-controller  Ingress cockroachdb-instance/sql-cockroachdb
  Normal  CREATE  28m                ingress-controller  Ingress cockroachdb-instance/sql-cockroachdb
  Normal  UPDATE  27m (x2 over 27m)  ingress-controller  Ingress cockroachdb-instance/sql-cockroachdb
  Normal  UPDATE  27m (x2 over 27m)  ingress-controller  Ingress cockroachdb-instance/sql-cockroachdb

If I try to port-forward the service that handles cockroachdb I get this error:

E0711 09:59:38.779413   43555 portforward.go:409] an error occurred forwarding 8081 -> 26258: error forwarding port 26258 to pod 8a774d302b846fdbdd7bbc6b3f35144d9712f15de9ab72d4ef9e0c8cdfa8ee85, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f1c36dc8-3647-e00e-b67e-7abb019b236b": read tcp4 127.0.0.1:33424->127.0.0.1:26258: read: connection reset by peer

This is the Kind: CrdbCluster - The installation of cockroachdb was done via kubectl apply with all default values.

apiVersion: crdb.cockroachlabs.com/v1alpha1
kind: CrdbCluster
metadata:
  # this translates to the name of the statefulset that is created
  name: cockroachdb
spec:
  dataStore:
    pvc:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: "20Gi"
        volumeMode: Filesystem
  resources:
    requests:
      # This is intentionally low to make it work on local k3d clusters.
      cpu: 500m
      memory: 2Gi
    limits:
      cpu: 2
      memory: 4Gi
  tlsEnabled: true
   ingress:
    #ui:
    #  ingressClassName: haproxy
    #  annotations:
    #    key: value
    #  host: cockroachdb.dev-service-stage.k8s.mi1.prod.cloudfire.it
    sql:
      ingressClassName: haproxy
      annotations:
        #key: value
      host: cockroachdb.dev-service-stage.k8s.mi1.prod.cloudfire.it
# You can set either a version of the db or a specific image name
# cockroachDBVersion: v23.1.4
  image:
    name: cockroachdb/cockroach:v23.1.4
  # nodes refers to the number of crdb pods that are created
  # via the statefulset
  nodes: 3
  additionalLabels:
    crdb: test

What tool are you using to create the cluster (minikube, kind, etc.)? — Alez, Jul 11 '23 at 07:31
I'm on openstack vms, i think you can consider this as baremetal installation using rancher2 — simone.benati, Jul 11 '23 at 07:37
Have you checked this [thread](https://stackoverflow.com/questions/44403931/how-can-i-connect-to-cockroachdb-from-outside-the-kubernetes-cluster) — Sai Chandra Gadde, Jul 11 '23 at 09:16

score 1 · Answer 1 · answered Jul 11 '23 at 08:36

1

The missing IP in the ingress list is because you have the error about the port-forward the service.

The reason you have that error, and the connection is refused, is there is no process listening on the port you're forwarding (26258) and this may be related either to:

you didn't change the default DB port to the port you want to forward (26258)
from the Dockerfile you're exposing the port 26258 even if your DB is still on the default port 26257

answered Jul 11 '23 at 08:36

Alez

1,913
3
18
22

Thank you for answering, I have the same issue with port-forwarding the service on 26257.. – simone.benati Jul 11 '23 at 08:40
Please show the Deployment config you're using or the Dockerfile if you changed the basic image – Alez Jul 11 '23 at 08:51
Added the Kind: CrdbCluster with my configs in the initial post – simone.benati Jul 11 '23 at 08:58
Ok, so you're using an operator. Revise the readme file https://github.com/cockroachdb/cockroach-operator/tree/master since every operator can be customized. To forward correctly, take a look of what ports are exposed by the service you have, for example for the DBConsole ```kubectl port-forward service/cockroachdb-public 8080``` To create an ingress refer to this issue https://github.com/cockroachdb/cockroach-operator/issues/76 and related docs. – Alez Jul 11 '23 at 09:16

score 1 · Answer 2 · answered Jul 13 '23 at 06:50

The issue was with metallb and a custom resource

while adding this to the manifests, everything was provisioned successfully:

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: advertisement
  namespace: metallb
spec:
  ipaddresspools:
  - first-pool

Kubernetes: Haproxy ingress isn't taking on any addresses

2 Answers2

Linked