Elastic Agent with Fleet Server in Docker fails with "Error: request to get security token from Kibana failed: Forbidden: %!w()"

Question

We are running all our test/staging/production Elastic workloads off Elastic Cloud, but for local development we use Docker for running Elastic, Kibana and APM Server.

APM Server has been flagged as deprecated, so we want to move to using Elastic Agent as Fleet Server, which is already how things are running in Elastic Cloud in the version we are at (7.17.8).

However, we recently change local configuration so we have until now been reapplying the entire local Elastic setup frequently using docker-compose and keeping everything locally without passwords.

Now we are trying to recreate this setup with Elastic Agent in a Fleet role, replacing the legacy apm-server. Here is the relevant yaml-config (note: ports are off by 1 deliberately, since in a transition period we actually need to run Elastic 7 and Elastic 8 side-by-side in dev-environments):

elasticsearch-v8:
  image: docker.elastic.co/elasticsearch/elasticsearch:8.6.0
  container_name: elasticsearch-v8
  networks:
    - network
  restart: always
  ports:
    - "9201:9201"
  environment:
    - http.port=9201
    - "discovery.type=single-node"
    - "bootstrap.memory_lock=false"
    - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    - "xpack.security.enabled=false"
    - "xpack.security.authc.api_key.enabled: true"

kibana-v8:
  image: docker.elastic.co/kibana/kibana:8.6.0
  container_name: kibana-v8
  networks:
    - network
  restart: always
  ports:
    - "5602:5602"
  depends_on:
    - elasticsearch-v8
  environment:
    SERVER_PORT: "5602"
    ELASTICSEARCH_HOSTS: '["http://elasticsearch-v8:9201"]'

fleet-server-v8:
  image: docker.elastic.co/beats/elastic-agent:8.6.0
  container_name: fleet-server-v8
  networks:
    - network
  restart: "always"
  ports:
    - "9243:9243"
  depends_on:
    - elasticsearch-v8
    - kibana-v8
  environment:
    FLEET_SERVER_ENABLE: true
    FLEET_SERVER_ELASTICSEARCH_HOST: "elasticsearch-v8:9201"
    FLEET_SERVER_PORT: 9243
    FLEET_SERVER_INSECURE_HTTP: true
    KIBANA_HOST: "http://kibana-v8:5602"
    KIBANA_FLEET_SETUP: true

apm-server-v8: # <--- This is our old way that still works, but we wish to replace it
  image: docker.elastic.co/apm/apm-server:8.6.0
  container_name: apm-server-v8
  networks:
    - network
  restart: always
  ports:
    - "8201:8200"
  depends_on:
    - elasticsearch-v8
    - kibana-v8
  command: >
    apm-server -e
      -E apm-server.rum.enabled=true
      -E setup.kibana.host=kibana-v8:5602
      -E apm-server.kibana.enabled=true
      -E apm-server.kibana.host=kibana-v8:5602
      -E output.elasticsearch.hosts=["elasticsearch-v8:9201"]
      -E apm-server.data_streams.wait_for_integration=false

When booting the fleet-server, we see the following in the log:

2023-01-25 10:58:19 Requesting service_token from Kibana.
2023-01-25 10:58:19 Error: request to get security token from Kibana failed: Forbidden: %!w()

Question: can we have the fleet-server boot without enabling xpack.security and creating a FLEET_SERVER_TOKEN first? We would rather not want developers having to go through any manual steps when reprovisioning local setup.

(xpack.security is disabled by default for localhost with no license)

Elastic Agent with Fleet Server in Docker fails with "Error: request to get security token from Kibana failed: Forbidden: %!w()"

0 Answers0