Connecting jaeger with elasticsearch backend storage on kubernetes cluster

Question

I have a kubernetes cluster on google cloud platform, and on it, I have a jaeger deployment via development setup of jaeger-kubernetes templates because my purpose is setup elasticsearch like backend storage, due to this, I follow the jaeger-kubernetes github documentation with the following actions

• I've created the services via production setup options

Here are configured the URLs to access to elasticsearch server and username and password and ports

kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/production-elasticsearch/configmap.yml

And here, there are configured the download of docker images of the elasticsearch service and their volume mounts.

kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/production-elasticsearch/elasticsearch.yml

And then, at this moment we have a elasticsearch service running over 9200 and 9300 ports

 kubectl get service elasticsearch                                                                                                                                [a89fbe2]
NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
elasticsearch   ClusterIP   None         <none>        9200/TCP,9300/TCP   1h

• I've follow with the creation of jaeger components using the kubernetes-jaeger production templates of this way:

---

λ bgarcial [~] → kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/jaeger-production-template.yml        
deployment.extensions/jaeger-collector created
service/jaeger-collector created
service/zipkin created
deployment.extensions/jaeger-query created
service/jaeger-query created
daemonset.extensions/jaeger-agent created

λ bgarcial [~/workspace/jaeger-elastic] at  master ?

According to the Jaeger architecture, the jaeger-collector and jaeger-query services require access to backend storage.

And so, these are my services running on my kubernetes cluster:

λ bgarcial [~/workspace/jaeger-elastic] at  master ?
→ kubectl get services                                                                                  [baefdf9]
NAME               TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                        AGE
elasticsearch      ClusterIP      None            <none>           9200/TCP,9300/TCP              3h
jaeger-collector   ClusterIP      10.55.253.240   <none>           14267/TCP,14268/TCP,9411/TCP   3h
jaeger-query       LoadBalancer   10.55.248.243   35.228.179.167   80:30398/TCP                   3h
kubernetes         ClusterIP      10.55.240.1     <none>           443/TCP                        3h
zipkin             ClusterIP      10.55.240.60    <none>           9411/TCP                       3h

λ bgarcial [~/workspace/jaeger-elastic] at  master ?

• I going to configmap.yml elastic search file kubectl edit configmap jaeger-configuration command in order to try to edit it in relation to the elasticsearch URLs endpoints (may be? ... At this moment I am supossing that this is the next step ...)

I execute it:

λ bgarcial [~] → kubectl edit configmap jaeger-configuration 

And I get the following edit entry:

 apiVersion: v1
data:
  agent: |
    collector:
      host-port: "jaeger-collector:14267"
  collector: |
    es:
      server-urls: http://elasticsearch:9200
      username: elastic
      password: changeme
    collector:
      zipkin:
        http-port: 9411
  query: |
    es:
      server-urls: http://elasticsearch:9200
      username: elastic
      password: changeme
  span-storage-type: elasticsearch
kind: ConfigMap
metadata:
  creationTimestamp: "2018-12-27T13:24:11Z"
  labels:
    app: jaeger
    jaeger-infra: configuration
  name: jaeger-configuration
  namespace: default
  resourceVersion: "1387"
  selfLink: /api/v1/namespaces/default/configmaps/jaeger-configuration
  uid: b28eb5f4-09da-11e9-9f1e-42010aa60002

Here ... do I need setup our own URLs to collector and query services, which will be connect wiht elasticsearch backend service?

How to can I setup the elasticsearch IP address or URLs here?

In the jaeger components, the query and collector need access to storage, but I don't know what is the elastic endpoint ...

Is this server-urls: http://elasticsearch:9200 a correct endpoint?

I am starting in the kubernetes and DevOps world, and I appreciate if someone can help me in the concepts and point me in the right address in order to setup jaeger and elasticsearch as a backend storage.

Answer 1

When you are accessing the service from the pod in the same namespace you can use just the service name. Example:

http://elasticsearch:9200

If you are accessing the service from the pod in the different namespace you should also specify the namespace. Example:

http://elasticsearch.mynamespace:9200
http://elasticsearch.mynamespace.svc.cluster.local:9200

To check in what namespace the service is located, use the following command:

kubectl get svc --all-namespaces -o wide

Note: Changing ConfigMap does not apply it to deployment instantly. Usually, you need to restart all pods in the deployment to apply new ConfigMap values. There is no rolling-restart functionality at the moment, but you can use the following command as a workaround:
(replace deployment name and pod name with the real ones)

kubectl patch deployment mydeployment -p '{"spec":{"template":{"spec":{"containers":[{"name":"my-pod-name","env":[{"name":"START_TIME","value":"'$(date +%s)'"}]}]}}}}'