How to configure Kibana for Swisscom elasticsearch public cloud (CloudFoundry) - docker

Question

Note: This question is specific to the Elasticsearch service provided by Swisscom
Question: (a.k.a: tl;dr)
What configuration is required to get the official Kibana docker container to connect to a Swisscom Elasticsearch Service?
Background:
Up until about a year ago the Swisscom public cloud offered a full ELK stack (Elasticsearch, Logstash, Kibana) in a single service offering. When this service was discontinued, Swisscom replaced it by just offering the Elasticsearch service and asked clients to setup their own Kibana and Logstash solutions via provided CloudFoundry build_packs (Kibana, Logstash). The migration recommendation was discussed here: https://ict.swisscom.ch/2018/04/building-the-elk-stack-on-our-new-elasticsearch/
More recently, the underlying OS (called "stack") that runs the applications on Swisscom's CloudFoundry-based PaaS offering, has been upgraded. The aforementioned build_packs are now outdated and have been declared as deprecated by Swisscom. The suggestion now is to move to a generic Docker container provided by Elastic as discussed here: https://github.com/swisscom/kibana-buildpack/issues/3
What I tried:
CloudFoundry generally works well with Docker containers and the whole thing should be as straight forward as providing some valid configuration to the docker container. My current manifest.yml for Kibana looks something like this, but the Kibana application ultimately fails to connect:
---
applications:
- name: kibana-test-example
docker:
image: docker.elastic.co/kibana/kibana:6.1.4
memory: 4G
disk_quota: 5G
services:
- elasticsearch-test-service
env:
SERVER_NAME: kibana-test
ELASTICSEARCH_URL: https://abcdefghijk.elasticsearch.lyra-836.appcloud.swisscom.com
ELASTICSEARCH_USERNAME: username_provided_by_elasticsearch_service
ELASTICSEARCH_PASSWORD: password_provided_by_elasticsearch_service
XPACK_MONITORING_ENABLED: true
Additional Info:
The Elasticsearch Service provided by Swisscom currently runs on version 6.1.3. As far as I'm aware it has x-pack installed.
What errors are you getting?
I played around with the configuration a bit and have seen different errors, most of which appear to be related to failing authentication against the Elasticsearch Service.
Here is some exemplary initial log output (seriously, though, you need a running Kibana just to be able to read that...)
2019-05-10T08:08:34.43+0200 [CELL/0] OUT Cell eda692ed-f4c3-4a5e-86aa-c0d1641b029f successfully created container for instance 385e5b7f-1570-46cd-532a-c5b4
2019-05-10T08:08:48.60+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:08:48Z","tags":["info","optimize"],"pid":6,"message":"Optimizing and caching bundles for graph, monitoring, apm, kibana, stateSessionStorageRedirect, timelion, login, logout, dashboardViewer and status_page. This may take a few minutes"}
2019-05-10T08:15:07.68+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:15:07Z","tags":["info","optimize"],"pid":6,"message":"Optimization of bundles for graph, monitoring, apm, kibana, stateSessionStorageRedirect, timelion, login, logout, dashboardViewer and status_page complete in 379.08 seconds"}
2019-05-10T08:15:07.77+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:kibana#6.1.4","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.82+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:elasticsearch#6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.86+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:xpack_main#6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.86+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:graph#6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.88+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:monitoring#6.1.4","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:xpack_main#6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:graph#6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:elasticsearch#6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
2019-05-10T08:15:11.39+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:15:11Z","tags":["reporting","warning"],"pid":6,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
2019-05-10T08:15:11.39+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:15:11Z","tags":["status","plugin:reporting#6.1.4","error"],"pid":6,"state":"red","message":"Status changed from uninitialized to red - Authentication Exception","prevState":"uninitialized","prevMsg":"uninitialized"}
The actually relevant error message seems to be this:
2019-05-10T08:15:11.66+0200 [APP/PROC/WEB/0] OUT {"type":"log","#timestamp":"2019-05-10T06:15:11Z","tags":["license","warning","xpack"],"pid":6,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}"}
I have tried to set XPACK_SECURITY_ENABLED: false as recommended elsewhere as well as setting the actual SERVER_HOST, which seemed to make things worse.
I would very much appreciate a working example from someone using the existing Kibana docker images to connect to the Swisscom-provided Elasticsearch Service.

Answer 1

Could it be that you confused username and password? When I check my service-key password comes before username, which might have lead to a copy-paste error on your side:
cf service-key myece mykey|grep kibana_system
"kibana_system_password": "aKvOpMVrXGCJ4PJht",
"kibana_system_username": "aksTxVNyLU4JWiQOE6V",
I tried pushing Kibana with your manifest.yml and it works perfectly in my case.
Swisscom has also updated the documentation on how to use Kibana and Logstash with Docker:
https://docs.developer.swisscom.com/service-offerings/kibana-docker.html
https://docs.developer.swisscom.com/service-offerings/logstash-docker.html