How to configure Kibana for Swisscom elasticsearch public cloud (CloudFoundry)

Question

Note: This question is specific to the Elasticsearch service provided by Swisscom

Question: (a.k.a: tl;dr)

What configuration is required to get the official Kibana docker container to connect to a Swisscom Elasticsearch Service?

Background:

Up until about a year ago the Swisscom public cloud offered a full ELK stack (Elasticsearch, Logstash, Kibana) in a single service offering. When this service was discontinued, Swisscom replaced it by just offering the Elasticsearch service and asked clients to setup their own Kibana and Logstash solutions via provided CloudFoundry build_packs (Kibana, Logstash). The migration recommendation was discussed here: https://ict.swisscom.ch/2018/04/building-the-elk-stack-on-our-new-elasticsearch/

More recently, the underlying OS (called "stack") that runs the applications on Swisscom's CloudFoundry-based PaaS offering, has been upgraded. The aforementioned build_packs are now outdated and have been declared as deprecated by Swisscom. The suggestion now is to move to a generic Docker container provided by Elastic as discussed here: https://github.com/swisscom/kibana-buildpack/issues/3

What I tried:

CloudFoundry generally works well with Docker containers and the whole thing should be as straight forward as providing some valid configuration to the docker container. My current manifest.yml for Kibana looks something like this, but the Kibana application ultimately fails to connect:

---
applications:
- name: kibana-test-example
  docker:
    image: docker.elastic.co/kibana/kibana:6.1.4
  memory: 4G
  disk_quota: 5G
  services:
    - elasticsearch-test-service
  env:
    SERVER_NAME: kibana-test
    ELASTICSEARCH_URL: https://abcdefghijk.elasticsearch.lyra-836.appcloud.swisscom.com
    ELASTICSEARCH_USERNAME: username_provided_by_elasticsearch_service
    ELASTICSEARCH_PASSWORD: password_provided_by_elasticsearch_service
    XPACK_MONITORING_ENABLED: true

Additional Info:

The Elasticsearch Service provided by Swisscom currently runs on version 6.1.3. As far as I'm aware it has x-pack installed.

What errors are you getting?

I played around with the configuration a bit and have seen different errors, most of which appear to be related to failing authentication against the Elasticsearch Service.

Here is some exemplary initial log output (seriously, though, you need a running Kibana just to be able to read that...)

   2019-05-10T08:08:34.43+0200 [CELL/0] OUT Cell eda692ed-f4c3-4a5e-86aa-c0d1641b029f successfully created container for instance 385e5b7f-1570-46cd-532a-c5b4
   2019-05-10T08:08:48.60+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:08:48Z","tags":["info","optimize"],"pid":6,"message":"Optimizing and caching bundles for graph, monitoring, apm, kibana, stateSessionStorageRedirect, timelion, login, logout, dashboardViewer and status_page. This may take a few minutes"}
   2019-05-10T08:15:07.68+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["info","optimize"],"pid":6,"message":"Optimization of bundles for graph, monitoring, apm, kibana, stateSessionStorageRedirect, timelion, login, logout, dashboardViewer and status_page complete in 379.08 seconds"}
   2019-05-10T08:15:07.77+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:kibana@6.1.4","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
   2019-05-10T08:15:07.82+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:elasticsearch@6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
   2019-05-10T08:15:07.86+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:xpack_main@6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
   2019-05-10T08:15:07.86+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:graph@6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
   2019-05-10T08:15:07.88+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:monitoring@6.1.4","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
   2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:xpack_main@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
   2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:graph@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
   2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:elasticsearch@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
   2019-05-10T08:15:11.39+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:11Z","tags":["reporting","warning"],"pid":6,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
   2019-05-10T08:15:11.39+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:11Z","tags":["status","plugin:reporting@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from uninitialized to red - Authentication Exception","prevState":"uninitialized","prevMsg":"uninitialized"}

The actually relevant error message seems to be this:

2019-05-10T08:15:11.66+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:11Z","tags":["license","warning","xpack"],"pid":6,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}"}

I have tried to set XPACK_SECURITY_ENABLED: false as recommended elsewhere as well as setting the actual SERVER_HOST, which seemed to make things worse.

I would very much appreciate a working example from someone using the existing Kibana docker images to connect to the Swisscom-provided Elasticsearch Service.

Why not migrating away from the Swisscom Cloud to another cloud service that knows his stuff? — Val, May 10 '19 at 06:37
Hahahaha - oh my - where do I start - we would need to meet in person for that discussion I think, but in short - our clients pay us for it and we can not change. — Chris, May 10 '19 at 06:43
Yeah, that's the only reason I could see for landing in there... I'm going to poke someone internally (at Elastic and Swisscom) and see if they can chime in. — Val, May 10 '19 at 06:47
I have the contact info of someone at Swisscom that you can contact to dig further into this (let me know how I can share this with you). For the benefit of others who are in the same situation as you are, please come back here and publish an answer when you have something working ;-) — Val, May 10 '19 at 07:39
Did you bind Kibana app with Elasticsearch service instance -- https://docs.developer.swisscom.com/service-offerings/elasticsearch.html#integrating-your-service ? — Jan Garaj, May 10 '19 at 20:10
Thanks @Val, I contacted our support contacts as well and they are working on making more documentation available. — Chris, May 13 '19 at 08:54

score 1 · Accepted Answer · answered May 13 '19 at 11:24

Could it be that you confused username and password? When I check my service-key password comes before username, which might have lead to a copy-paste error on your side:

cf service-key myece mykey|grep kibana_system
 "kibana_system_password": "aKvOpMVrXGCJ4PJht",
 "kibana_system_username": "aksTxVNyLU4JWiQOE6V",

I tried pushing Kibana with your manifest.yml and it works perfectly in my case.

Swisscom has also updated the documentation on how to use Kibana and Logstash with Docker:

https://docs.developer.swisscom.com/service-offerings/kibana-docker.html https://docs.developer.swisscom.com/service-offerings/logstash-docker.html

Thanks, that's great info - I'll read through the new documentation. I did notice the username and password to be in sorted order and can pretty much rule out the copy & paste error, since the log shows which username could not be authenticated and I must have confirmed that a million times ;) — Chris, May 13 '19 at 12:44

How to configure Kibana for Swisscom elasticsearch public cloud (CloudFoundry)

1 Answers1