0

I have a single container on elastic beanstalk. It's a .net core 2.2 application deployed with a docker container running microsoft/dotnet:2.2-aspnetcore-runtime.

I've been trying to setup logging to cloudwatch, and I went into Elastic Beanstalk -> Environments -> my-environment -> Configuration and enabled "log streaming" (rotate logs is disabled)

I could now see my log's in cloudwatch, and I can see the log statements from my application in: /aws/elasticbeanstalk/my-environment/var/log/eb-docker/containers/eb-current-app/stdouterr.log

The problem is, that they are never updated. The only time more data is put into the logs, is if I redeploy the application (even after 12H, there's still nothing new in the logs).

I've tried adding a file: .ebextensions/logs.config with the content:

option_settings:
- namespace: aws:elasticbeanstalk:cloudwatch:logs
option_name: StreamLogs
value: true
- namespace: aws:elasticbeanstalk:cloudwatch:logs
option_name: DeleteOnTerminate
value: false
- namespace: aws:elasticbeanstalk:cloudwatch:logs
option_name: RetentionInDays
value: 7

But it's the same. The content in the logs is only updated when I redeploy the application. Secondly, I followed this: Elastic Beanstalk Single Container Docker - use awslogs logging driver

option_settings:
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: StreamLogs
    value: true
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: DeleteOnTerminate
    value: false
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: RetentionInDays
    value: 14

files:
  "/etc/awslogs/config/stdout.conf":
    mode: "000755"
    owner: root
    group: root
    content: |
      [/var/log/containers/docker-stdout]
      log_group_name=/aws/elasticbeanstalk/`{ "Ref" : "AWSEBEnvironmentName" }`/docker-stdout.log
      log_stream_name={instance_id}
      file=/var/log/containers/*-stdouterr.log

commands:
  "00_restart_awslogs":
    command: service awslogs restart

What am I missing? There isn't a lot of traffic on the site, but shouldn't the logs be updated at least every 5 minutes or so? I've been reading about file_fingerprint_lines and setting awslogs as driver on docker, but can't seem to get any of that to work.

I don't have much experience with EBS or Cloudwatch, so any help is highly appreciated. One of the problems is, that I don't really know where to look for debugging this

user826988
  • 83
  • 7
  • Have you seen [this](https://stackoverflow.com/questions/61201006/elastic-beanstalk-log-task-customization-on-amazon-linux-2-platforms/61995539#61995539) question? Also do you run single-docker or multi-docker environmnet on EB? – Marcin Aug 09 '20 at 00:16
  • Thanks, i haven't seen that answer before. I'm going to try it out tonight. It's a single docker instance. – user826988 Aug 09 '20 at 11:04
  • I've made a bit progress, or at least, I've got some pointers on what's wrong. I SSH'ed into the instance, and started looking at logs. I can see that the stderr log is updating, but I get the following error in my awslogs.log: cwlogs.push.stream - WARNING - 6563 - Thread-1 - No file is found with given path '/var/log/containers/*-stdouterr.log'. there is no /var/log/containers/ folder, so I'll will look into how to change it to look the correct place tomorrow again – user826988 Aug 09 '20 at 22:20

1 Answers1

0

I got this solved, by having this in my .config file:

option_settings:
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: StreamLogs
    value: true
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: DeleteOnTerminate
    value: false
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: RetentionInDays
    value: 14

files:
  "/etc/awslogs/config/stdout.conf":
    mode: "000755"
    owner: root
    group: root
    content: |
      [/var/log/containers/docker-stdout]
      log_group_name=/aws/elasticbeanstalk/`{ "Ref" : "AWSEBEnvironmentName" }`/docker-stdout.log
      log_stream_name={instance_id}
      file=/var/log/eb-docker/containers/eb-current-app/*-stdouterr.log

The answer itself is actually not as interesting, as how I got it debugged (posted here in case it will help others):

  1. SSH into the instance by using eb ssh
  2. Found the log: /var/log/awslogs.log and saw the error: cwlogs.push.stream - WARNING - 6563 - Thread-1 - No file is found with given path '/var/log/containers/*-stdouterr.log'
  3. I followed the path (/var/log/containers) only to find that the folder structure did not exist. From the logs that was send to cloudwatch, I could see the path was: /var/log/eb-docker/containers/eb-current-app/ which I checked, and found the log files.
  4. From some of the other questions that I've been reading, I could see that the configuration for awslogs (which is the agent sending output to aws) was located in: /etc/awslogs/config/stdout.conf. Looking in this file, I saw the path /var/log/containers/*-stdouterr.log
  5. I changed the path to /var/log/eb-docker/containers/eb-current-app/*-stdouterr.log and restarted the awslogs service sudo service awslogs restart

Now I could see the log being updated. Why it did send logs to cloudwatch on deploy, I don't know, but the above fixed the issue

user826988
  • 83
  • 7