9

I have a web service deployed to an Elastic Beanstalk environment running the Docker Multi-Container stack. I have enabled Log Streaming to CloudWatch on the environment, so about five different log groups show up in Cloudwatch, and so when I click "Request Logs" from Beanstalk it loads a webpage that shows me all the log files, one after another. I've noticed that there are some logs on this web page that do not show up as Log Groups in CloudWatch, and these are the logs I really care about. My question is how do I get them to show up as CloudWatch Log Groups?

In particular, the five Log Groups that Elastic Beanstalk automatically created for me are:

  • /aws/elasticbeanstalk/my-web-service/var/log/docker-events.log
  • /aws/elasticbeanstalk/my-web-service/var/log/eb-activity.log
  • /aws/elasticbeanstalk/my-web-service/var/log/eb-ecs-mgr.log
  • /aws/elasticbeanstalk/my-web-service/var/log/ecs/ecs-agent.log
  • /aws/elasticbeanstalk/my-web-service/var/log/ecs/ecs-init.log

And when I look in the file that gets generated when I "request logs," those five are indeed there. But these other log files are also represented:

  • /aws/elasticbeanstalk/my-web-service/var/log/awslogs.log
  • /aws/elasticbeanstalk/my-web-service/var/log/docker
  • /aws/elasticbeanstalk/my-web-service/var/log/docker-ps.log
  • /aws/elasticbeanstalk/my-web-service/var/log/eb-commandprocessor.log
  • /aws/elasticbeanstalk/my-web-service/var/log/containers/my-svc-8edcf9cec583-stdouterr.log

It's that last one that I'm really interested in, the one ending in stdouterr.log. That's where my containerized application writes all of its log messages to. What I would like to see is a Log Group in CloudWatch that corresponds to that stdouterr.log file. As far as I can tell, the 12-digit ID that's in the log file name is the ID of the docker image that gets installed on the host, and is subject to change every time you restart the server. So I'm guessing I'll likely need to mount a volume, or something like that, in the Dockerrun.aws.json configuration? And furthermore I would guess that I'd then need to manually add a Log Group to CloudWatch? How can I get this file to show up?

soapergem
  • 9,263
  • 18
  • 96
  • 152

1 Answers1

10

It looks like you currently only have the default logs being sent to Cloudwatch logs. You can add additional logs to the cloudwatch agent through your .ebextensions 

### BEGIN .ebextensions/logs.config
option_settings:
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: StreamLogs
    value: true
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: DeleteOnTerminate
    value: false
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: RetentionInDays
    value: 7

files:
  "/etc/awslogs/config/stdout.conf":
    mode: "000755"
    owner: root
    group: root
    content: |
      [docker-stdout]
      log_group_name=/aws/elasticbeanstalk/environment_name/docker-stdout
      log_stream_name={instance_id}
      file=/var/log/eb-docker/containers/eb-current-app/*-stdouterr.log

commands:
  "00_restart_awslogs":
    command: service awslogs restart

### END .ebextensions/logs.config

source

Stephen
  • 3,607
  • 1
  • 27
  • 30
  • I finally got around to trying this but sadly it had no effect. Specifically, I zipped together my Dockerrun.aws.json file with the .ebextensions folder containing a file just like this one. However no new log groups showed up in Cloudwatch. Any more ideas? – soapergem Apr 13 '18 at 00:51
  • I will also note that if I SSH into the EC2 instance that backs EB, I can see this config file (`/etc/awslogs/config/stdout.conf`) did indeed get created. However it's seemingly not acting on it. – soapergem Apr 13 '18 at 04:15
  • 1
    Have you checked the agent logs `/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log` maybe it will give you more info to what is failing – Stephen Apr 13 '18 at 07:04
  • There is no `/opt/aws/amazon-cloudwatch-agent` directory – soapergem Apr 13 '18 at 14:58
  • 3
    Figured it out - the instance profile role didn't have permission to push to Cloudwatch. When I added that, this started working. Thanks! – soapergem Apr 13 '18 at 18:13
  • 2
    Always permissions! :) – Stephen Apr 13 '18 at 19:57
  • Ok, one other thing that's weird - although it created the log group successfully, it set the wrong retention date on it. It seems to be ignoring what I have in the file (i.e. 7 days) and doing Unlimited instead. – soapergem Apr 13 '18 at 20:55
  • Does your profile permission include `logs:putRetentionPolicy`? – Stephen Apr 13 '18 at 21:15
  • @soapergem What was the policy you had to add? `CloudWatchLogsFullAccess`? – kierans Sep 29 '19 at 12:54