How to use the elastic-agent container to collect service logs from another container?

Question

My target container contains NGINX logs which I wanted to collect from Elastic Fleet's NGINX Integration.

I followed every step, even successfully hosting the fleet server and the agent in two separate containers, what confuses me, is how can I configure my Agent which has the NGINX integration setup on its policy, to collect logs from the service container?

I have mostly encountered examples using the elastic-agent as a package installer directly on the target container.

I've attached three snippets of my docker-compose setup, that I follow for the Fleet, Agent and App containers.

FLEET SERVER

  fleet:
    image: docker.elastic.co/beats/elastic-agent:$ELASTIC_VERSION
    healthcheck:
      test: "curl -f http://127.0.0.1:8220/api/status | grep HEALTHY 2>&1 >/dev/null"
      retries: 12
      interval: 5s
    hostname: fleet
    container_name: fleet
    restart: always
    user: root
    environment:
    - FLEET_SERVER_ENABLE=1
    - "FLEET_SERVER_ELASTICSEARCH_HOST=https://elasticsearch:9200"
    - FLEET_SERVER_ELASTICSEARCH_USERNAME=elastic
    - FLEET_SERVER_ELASTICSEARCH_PASSWORD=REPLACE1
    - FLEET_SERVER_ELASTICSEARCH_CA=$CERTS_DIR/ca/ca.crt
    - FLEET_SERVER_INSECURE_HTTP=1
    - KIBANA_FLEET_SETUP=1
    - "KIBANA_FLEET_HOST=https://kibana:5601" 
    - KIBANA_FLEET_USERNAME=elastic
    - KIBANA_FLEET_PASSWORD=REPLACE1
    - KIBANA_FLEET_CA=$CERTS_DIR/ca/ca.crt
    - FLEET_ENROLL=1
    ports:
      - 8220:8220
    networks:
      - elastic
    volumes:
       - certs:$CERTS_DIR

Elastic Agent

agent:
    image: docker.elastic.co/beats/elastic-agent:$ELASTIC_VERSION
    container_name: agent
    hostname: agent
    restart: always
    user: root 
    healthcheck:
      test: "elastic-agent status"
      retries: 90
      interval: 1s
    environment:
      - FLEET_ENROLLMENT_TOKEN=REPLACE2
      - FLEET_ENROLL=1
      - FLEET_URL=http://fleet:8220
      - FLEET_INSECURE=1
      - ELASTICSEARCH_HOSTS='["https://elasticsearch:9200"]'
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=REPLACE1
      - ELASTICSEARCH_CA=$CERTS_DIR/ca/ca.crt
      - "STATE_PATH=/usr/share/elastic-agent"
    networks:
      - elastic
    volumes:
       - certs:$CERTS_DIR

App Container (NGINX logs)

  demo-app:
    image: ubuntu:bionic
    container_name: demo-app
    build:
      context: ./docker/
      dockerfile: Dockerfile
    volumes:
      - ./app:/var/www/html/app
      - ./docker/nginx.conf:/etc/nginx/nginx.conf
    ports:
      - target: 90
        published: 9090
        protocol: tcp
        mode: host
    networks:
      - elastic

The ELK stack currently runs on version 7.17.0. If anyone could provide any info on what next needs to be done , It'll be very much helpful, thanks!

I believe the Elastic Agent has to be deployed on the same `host` as the service it is monitoring. Elastic Agent are reading the logs files, they have to have access to those. — Paulo, Feb 20 '22 at 15:44
Hey @Paulo that's what I had also thought, since they already had a package installer out there, but since they offered an alternative using the elastic-agent image, I assumed there should be a way, but unfortunately they haven't broken down the necessary steps to follow through this approach, unlike using their package installer. — , Feb 20 '22 at 17:37
Looks like Alphine Linux images don't seem to install the package installer, which means the elastic-agent image is my only choice. https://discuss.elastic.co/t/unable-to-install-fleet-managed-elastic-agent/274774/4?u=gibz — , Feb 21 '22 at 06:52
This means you have to install your server on this base image, I believe. — Paulo, Feb 21 '22 at 07:30

score 0 · Answer 1 · answered May 20 '22 at 14:33

you could share nginx log files through volume mount. mount a directory to nginx log directory, and mount that to a directory in your elastic agent container. then youre good to harvest the nginx log in elastic agent container from there.

there might be directory read write permission problem, feel free to ask below.

kinda like:

nginx compose:

   demo-app:
     ...
     volumes:
       - ./app:/var/www/html/app
       - ./docker/nginx.conf:/etc/nginx/nginx.conf
+      - /home/user/nginx-log:/var/log/nginx/access.log
     ...

elastic agent compose:

 services:
   agent:
     ...
     volumes:
        - certs:$CERTS_DIR
+       - /home/user/nginx-log:/usr/share/elastic-agent/nginx-log

How to use the elastic-agent container to collect service logs from another container?

1 Answers1