Log streaming and premissions with .ebextensions

Question

I'm trying to stream a log file my application writes. I used https://github.com/awsdocs/elastic-beanstalk-samples/blob/master/configuration-files/aws-provided/instance-configuration/logs-streamtocloudwatch-linux.config as a starting point. Unfortunately it seems that /etc/awslogs is no longer the correct path to add additional CloudWatch config.

I found out that I can place the file in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/my.json to make it work. But I'm unable to create this file and make CloudWatch parse it using .ebextensions. I tried to do this using a file with the following contents. The file does not exist after deployment. If I used the filename proposed by awsdocs the file was created.

.ebextensions/02-logs-streamtocloudwatch.config:

files:
  "/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/my.json" :
    mode: "000644"
    owner: root
    group: root
    content: |
        {
            "logs": {
                "logs_collected": {
                    "files": {
                        "collect_list": [
                            {
                                "file_path": "/var/log/eb-docker/containers/eb-current-app/my.log",
                                "log_group_name": "/aws/elasticbeanstalk/my-env/var/log/eb-docker/containers/eb-current-app/my.log",
                                "log_stream_name": "{instance_id}"
                            }
                        ]
                    }
                }
            }
        }

Also, my log file should have 666 permissions for now (I know that that's not the idea). Creating a file in /opt/elasticbeanstalk/hooks/appdeploy/post/99_permissions.sh containing the following content using .ebextensions does not seem to work either.

.ebextensions/04-permissions.config:

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/99_dbg_permissions.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      chmod 666 /var/log/eb-docker/containers/eb-current-app/dbg.log

What am I missing about how to use .ebextensions and/or the CloudWatch config in Beanstalk?

I would like to not only fix this, but understand the problem (where can I create/modify files when/how). I appreciate any help...

". But I'm unable to create this file and make CloudWatch parse it using .ebextensions." why? How did you try doing that? What errors do you get? — Marcin, Aug 17 '20 at 23:06
@Marcin: I added the files I used to my post. A big part of my problem is that I don't get why I seem to be able to create files and change permissions in some locations, but not everwhere. I'm guessing that I run into some kind of timing problem (using the wrong hook), maybe..? — ritt0r, Aug 18 '20 at 19:39

score 4 · Answer 1 · answered Aug 19 '20 at 13:11

Thanks to @Marcin's answer I was able to add my log file to the log streaming config:

.platform/hooks/postdeploy/02-logs-streamtocloudwatch.sh

#!/bin/bash

echo '{
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/var/log/eb-docker/containers/eb-current-app/my.log",
                        "log_group_name": "/aws/elasticbeanstalk/my-env/var/log/eb-docker/containers/eb-current-app/my.log",
                        "log_stream_name": "{instance_id}"
                    }
                ]
            }
        }
    }
}' > "/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/my.json"
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start

score 0 · Accepted Answer · answered Aug 18 '20 at 22:58

The folder /opt/elasticbeanstalk/hooks/ does not exist, as you correctly noted. The reason is that you are probably using Amazon Linux 2 (AL2) based EB environment, instead of AL1. The folder was only available in the old EBs based on AL1.

On AL2, there is new hooks mechanism as described in Application deployment platform hooks.

Therefore, you have to adapt your code to use the new hooks, or switch back to EB environment based on AL1.

Log streaming and premissions with .ebextensions

2 Answers2

Linked