9

I want to setup three containers, one for logstash, one for elasticsearch and one for kibana. The last two are fine as the are but I need to configure the first one so it has and uses http input plungin and then to work with the CSV I'm going to pass it.

So far I've tried this, it runs but I think that it's not using the configurarion I tell it

    version: '3.3'
services:
  logstash:
    image: docker.elastic.co/logstash/logstash:6.7.0
#    configs:
#    - source: logstash_config
#      target: /etc/logstash/conf.d/logstash.conf
#    command: bash -c "logstash -f /etc/logstash/conf.d/logstash.conf && bin/logstash-plugin install logstash-input-http"  
    command: bash -c 'bin/logstash -e "input { http { } } output { stdout { codec => rubydebug} }" && bin/logstash-plugin install logstash-input-http'
    links:
      - elasticsearch
    ports:
      - 5044:5044
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.0
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9200:9200"
  kibana:
    image: docker.elastic.co/kibana/kibana:6.7.0
    ports:
      - "5601:5601"
   
configs:
  logstash_config:
    file: ./configs/logstash.conf
      
volumes:
  esdata1:
    driver: local

The configuration so far is (It still does not have the csv part)

    input {
  http {
    port => 8080
    ssl => off
 }
}
output {
  elasticsearch {
    hosts => "127.0.0.1"
    codec => "json"
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

Any idea on how to make logstash use the http input plugin with docker compose???

Thanks in advance.

anothermh
  • 9,815
  • 3
  • 33
  • 52
RaistlinMolina
  • 345
  • 1
  • 9
  • You need to change the port in your docker config, you are exposing the port 5044, but your input is listening on the port 8080. – leandrojmp Dec 11 '19 at 22:16

2 Answers2

2

The container uses port 8080, so you need to forward the port from the host to the network interface of the container. In this example 32000 is forwarded to 8080.

  logstash:
# blabla
    ports:
      - "5044:5044"
      - "9600:9600"
      - "32000:8080"

docker-compose -f .\docker-compose.yml up -d

netstat -ant | findstr 32000
  TCP    0.0.0.0:32000 0.0.0.0:0 LISTENING

So docker starts a socketserver listening on 32000 and forwards it to docker's bridged ip port 8080 where the http input plugin is listening.

Now sending a http request to 32000 comes in at the filter which logs it to elasticsearch if that is the output plugin.

curl -X PUT 'http://ip:32000' -d 'insert log here'

    GET .myindex/_search
    {
      "query": {
        "query_string": {
          "query": "insert",
          "fields": ["message"]
        }
      }
    }

      {
        "_index": ".myindex",
        "_id": "ertqY4IBbsVkN0caS8Ml",
        "_score": 1,
        "_source": {
          "@timestamp": "2022-08-03T11:14:46.570619Z",
          "message": "insert log here",
          "http": {
            "request": {
              "mime_type": "application/x-www-form-urlencoded",
              "body": {
                "bytes": "15"
.........
Serve Laurijssen
  • 9,266
  • 5
  • 45
  • 98
0

The "configs" element in docker-compose is only applicable if you are running Docker in Swarm mode. I suggest you mount your config file as a volume instead.

You do not need to install the input-http plugin : it is already bundled in the docker image.

And finally, if you want to be able to make a connection to logstash on port 8080, you need to map the port so that it is accessible from outside the containers.

This would be the resulting service in your docker-compose file :

services:
  logstash:
    image: docker.elastic.co/logstash/logstash:6.7.0
    command: logstash -f /etc/logstash/conf.d/logstash.conf
  links:
    - elasticsearch
  ports:
    - 5044:5044
    - 8080:8080
  volumes:
    - ./configs/logstash.conf:/etc/logstash/conf.d/logstash.conf:ro

There is also a mistake in your config file : if logstash tries to connect to Elasticsearch using 127.0.0.1, it will land on itself (the logstash container). Use "elasticsearch" instead : it will resolve to the internal ip of the ES container.

input {
  http {
    port => 8080
    ssl => off
 }
}
output {
  elasticsearch {
    hosts => "elasticsearch"
    codec => "json"
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

Cheers !

lbndev
  • 781
  • 6
  • 14