Cannot see Log information in Kibana running in Docker Container in my Spring Boot App (ELK)

Question

I have an issue to show logger information Kibana running in Docker Container in my Spring Boot App. After I ran this command(docker-compose -d), all containers covering ELK ran in Docker. I opened Kibana with this url (http://localhost:5601). Then I created an index related to Logstash. I couldn't see any logger related with my project information in Kibana.

Here is the screenshot shown below.

How can I fix it?

Here is my logstash.conf file shown below.

    input {
        tcp {
                port => 5000
        }
        file {
                path => "C:/Users/{username}/IdeaProjects/SpringBootElk/Springboot-Elk.log"
                sincedb_path => "/dev/null"
                start_position => "beginning"
        }
}
output {
        stdout{
                codec => rubydebug
        }
        elasticsearch {
                hosts => "elasticsearch:9200"
        }
}

Here is logstash conf in docker-compose.yml

services:
  logstash:
    image: docker.elastic.co/logstash/logstash:7.15.2
    user: root
    command: -f /etc/logstash/conf.d/
    volumes:
      - ./elk/logstash/:/etc/logstash/conf.d/
      - ./Users/{username}/IdeaProjects/SpringBootElk/Springboot-Elk.log:/Springboot-Elk.log
    ports:
      - "5000:5000"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    depends_on:
      - elasticsearch

Here is my Project File : My Project

In addition to `start_position`, can you also add `sincedb_path => "/dev/null"` and see if it goes better? You might also need to map `/Users/...` as a docker volume otherwise Logstash will not see your local drive — Val, Apr 14 '22 at 11:48
Yes, but also check the volume mapping as I think it's the reason why it's not working, your docker container doesn't have access to your local `/Users` folder — Val, Apr 14 '22 at 12:14
@Val I cannot see any logs coming from Springboot-Elk.log in Kibana. I think this is my issue. I couldn't fix it. How can I do that? Is it possible to look through my code? — S.N, Apr 14 '22 at 12:16
@Val How can I reach the log file? Do I delete "file" tag in logstash.conf? — S.N, Apr 14 '22 at 12:18
@Val How can I define log in volume of logstash in docker-compose.yml? Is it right - ./Users/{username}/IdeaProjects/SpringBootElk/Springboot-Elk.log:/Springboot-Elk.log in volume? — S.N, Apr 14 '22 at 12:31

Val · Accepted Answer · 2022-04-14T14:02:56.857

3

You probably need to add another volume in your docker-compose configuration in order to map the C:/Users/... folder of your local host to the Docker container, otherwise Logstash running inside a Docker container will not be able to see your filesystem.

volumes:
  - ./elk/logstash/:/etc/logstash/conf.d/
  - "C:/Users/{username}/IdeaProjects/SpringBootElk:/tmp/logs"

Note: if it doesn't work for your Windows environment, you might want to check this thread.

Then you can modify your file input like this:

file {
        path => "/tmp/logs/*"
        sincedb_path => "/dev/null"
        start_position => "beginning"
}

UPDATE: So you have two modes...

you run the app from IntelliJ and the logs are produced on your localhost and it works
you run the app from docker-compose and the logs are produced in the app's container and Logstash which runs in another container doesn't see them

In this case, you need to share a volume between containers. Below, we define a shared volume called app-logs. The app container will produce logs in /path/to/Springboot-Elk.log (you might need to adjust the path according to your application configuration) and the logstash container will see them in /tmp/logs.

So now your Logstash container is able to read both the logs produced by your application running on localhost and from your application running from its own container.

services:
  logstash:
    ...
    volumes:
      - ./elk/logstash/:/etc/logstash/conf.d/
      - "C:/Users/{username}/IdeaProjects/SpringBootElk:/tmp/logs"
      - app-logs:/tmp/logs
    ...
  app:
    ...
    volumes:
      - app-logs:/path/to/Springboot-Elk.log
    ...

volumes:
  app-logs:

edited Apr 14 '22 at 14:02

answered Apr 14 '22 at 12:36

Val

207,596
13
358
360

It works. One more question to you. How can I write log in Springboot-Elk.log in Docker? I cannot see any log when I work in the docker. – S.N Apr 14 '22 at 12:48
I couldn't see any new logs in Springboot-Elk.log when I work in Docker. When I work in localhost, logs info is written in the log file. How can I fix it? I only see logs coming from the process of localhost in Kibana. – S.N Apr 14 '22 at 13:08
I'd say that your volume mapping definition is not correct then, you should double check it. You have a Windows system so `./Users` would not be correct, check my answer again – Val Apr 14 '22 at 13:13
Do I use this code "./Users/{username}/IdeaProjects/SpringBootElk:/tmp/logs" in volume? – S.N Apr 14 '22 at 13:17
Why don't you check my answer, it shows what you should use – Val Apr 14 '22 at 13:27
Where do I use ./Users? I updated my repository. – S.N Apr 14 '22 at 13:31
It didn't work. – S.N Apr 14 '22 at 22:05
Your [docker-compose](https://github.com/Rapter1990/SpringBootElk/blob/master/docker-compose.yml) file doesn't reflect what I suggested in my answer – Val Apr 15 '22 at 03:41
I defined logback.xml but I cannot see logs from Controller and Service in the console. Where is the problem? – S.N Apr 15 '22 at 08:43

Cannot see Log information in Kibana running in Docker Container in my Spring Boot App (ELK)

1 Answers1