11

I'm wondering how to do log task customization in the new Elastic Beanstalk platform (the one based on Amazon Linux 2). Specifically, I'm comparing:

  • Old: Single-container Docker running on 64bit Amazon Linux/2.14.3
  • New: Single-container Docker running on 64bit Amazon Linux 2/3.0.0

(My question actually has nothing to do with Docker as such, I'm speculating the problem exist for any of the new Elastic Beanstalk platforms).

Previously I could follow Amazon's recipe, meaning put a file into /opt/elasticbeanstalk/tasks/bundlelogs.d/ and it would then be acted upon. This is no longer true.

Has this changed? I can't find it documented. Anyone been successful in doing log task customization on the newer Elastic Beanstalk platform? If so, how?

Minimal working example

I've created a minimal working example and deployed on both platforms.

Dockerfile:

FROM ubuntu
COPY daemon-run.sh /daemon-run.sh
RUN chmod +x /daemon-run.sh
EXPOSE 80
ENTRYPOINT ["/daemon-run.sh"]

Dockerrun.aws.json:

{
  "AWSEBDockerrunVersion": "1",
  "Logging": "/var/mydaemon"
}

daemon-run.sh:

#!/bin/bash
echo "Starting daemon" # output to stdout
mkdir -p /var/mydaemon/deeperlogs
while true; do
   echo "$(date '+%Y-%m-%dT%H:%M:%S%:z')  Hello World" >> /var/mydaemon/deeperlogs/app_$$.log
   sleep 5
done

.ebextensions/mydaemon-logfiles.config:

files: 
  "/opt/elasticbeanstalk/tasks/bundlelogs.d/mydaemon-logs.conf" :
    mode: "000755"
    owner: root
    group: root
    content: |
       /var/log/eb-docker/containers/eb-current-app/deeperlogs/*.log

If I do "Full Logs" action on the old platform I would get a ZIP with my deeperlogs included inside var/log/eb-docker/containers/eb-current-app. On the new platform I don't.

Investigation

If you look on the disk you'll see that the new Elastic Beanstalk doesn't have a /opt/elasticbeanstalk/tasks folder at all, unlike the old one. Hmm.

lbruun
  • 241
  • 1
  • 6
  • You can create log tasks using Amazon Linux 2 `.platform/hooks/`, as described in this [answer](https://stackoverflow.com/a/65202757). Based on the [Amazon Linux 2 migration docs](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.migration-al.html), I believe this is preferred over using `.ebextensions`. – djvg Jan 08 '21 at 09:47
  • @djvg. Whether or not fiddling with the log task configuration is done via `.ebextensions` or via platform hooks (now preferred as you mention) is an implementation detail of this question. The issue was that - up until the Aug 2020 release of the AL2 platforms - it simply wasn't possible to do log task customization for EB AL2 platforms the way it was documented by Amazon. Now it [has been fixed](https://stackoverflow.com/a/63278952/1504556) and you can fiddle with the files in `/opt/elasticbeanstalk/tasks` using the method of your choice (yes, hooks method preferred). – peterh Jan 08 '21 at 14:11
  • @peterh: Thanks for the explanation. That is more or less what I understood from the question and the accepted answer below. I just thought I would mention the platform hooks option, as that had not been mentioned anywhere on this page yet. – djvg Jan 08 '21 at 14:23

3 Answers3

3

On Amazon Linux 2 the folder is:

/opt/elasticbeanstalk/config/private/logtasks/bundle

The .ebextensions/mydaemon-logfiles.config should be:

files: 
  "/opt/elasticbeanstalk/config/private/logtasks/bundle/mydaemon-logs.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
       /var/mydaemon/deeperlogs/*.log

container_commands:
  append_deeperlogs_to_applogs:
    command: echo -e "\n/var/log/eb-docker/containers/eb-current-app/deeperlogs/*" >> /opt/elasticbeanstalk/config/private/logtasks/bundle/applogs

The mydaemon-logfiles.config also adds deeperlogs into applogs file. Without it deeperlogs will not be included in the download log zip bundle. Which is intresting, because the folder will be in the correct location, i.e., /var/log/eb-docker/containers/eb-current-app/deeperlogs/. But without being explicitly listed in applogs, it will be skipped when zip bundle is being generated.

I tested it with single docker environment (3.0.1).

The full log bundle successful contained deeperlogs with correct log data:

enter image description here

enter image description here

Hope that this will help. I haven't found any references for that. AWS documentaiton does not document this, as it is mostly based on Amazon Linux 1, not Amazon Linux 2.

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • Does this work for the PHP platform too? I tried this approach and it doesn't seem to work here. No custom logs are published in CloudWatch. Only the default ones. Unfortunately, the AWS documentation is really bad here... – Alexander Barton Jun 07 '20 at 10:56
  • @AlexanderBarton Its for docker env. But I guess it could be modified to work on other environments as well. – Marcin Jun 07 '20 at 11:05
  • If I only knew how to make this working again. On Amazon Linux 1, I had some boilerplate code ready to publish custom logs. But it has changed and I'm trying to figure it out for hours now. If I find a solution, I will share it :) – Alexander Barton Jun 07 '20 at 15:41
  • 2
    Thanks for this. It's extremely frustrating that Amazon's "latest" docs refer to outdated information – Andy Jun 26 '20 at 09:50
  • @Andy Yes. I feel your pain. – Marcin Jun 26 '20 at 10:14
  • This answer now only applies to Elastic Beanstalk AL2-based platforms released **prior** to 04-AUG-2020. See my answer [here](https://stackoverflow.com/a/63278952/2282938). – lbruun Aug 06 '20 at 07:43
  • @lbruun Thanks for letting me know. But your question is from may and my answer well. Not August. – Marcin Aug 06 '20 at 07:46
  • I agree. Your answer was correct back then which is why the bounty was awarded to you. :-) – lbruun Aug 06 '20 at 07:57
0

Amazon has fixed this problem in version of the Elastic Beanstalk AL2 platforms released on 04-AUG-2020.

It has been fixed so that log task customization on AL2-based platforms now works the way it has always worked (i.e. on the prevision generation AL2018 platforms) and you can therefore follow the official documentation in order to make this happen.

Succesfully tested with platform "Docker running on 64bit Amazon Linux 2/3.1.0". If you (still) use "Docker running on 64bit Amazon Linux 2/3.0.x" then you must use the undocumented workaround described in Marcin's answer but you are probably better off by upgrading your platform version.

lbruun
  • 241
  • 1
  • 6
0

As of 2021/11/05, I tried the accepted answer and various other examples including the latest official documentation on using the .ebextensions folder with *.config files without success.

Most likely something I was doing wrong but here's what worked for me. The version I'm using: Docker running on 64bit Amazon Linux 2/3.4.8

Simply, add a volume to your docker-compose.yml file to share your application logs to the Elastic Beanstalk log directory.

Example docker-compose.yml:

version: "3.9"
services:
  app:
    build: .
    ports:
      - "80:80"
    user: root
    volumes:
      - ./:/var/www/html
      # "${EB_LOG_BASE_DIR}/<service name>:<log directory inside container>
      - "${EB_LOG_BASE_DIR}/app:/var/www/html/application/logs" # ADD THIS LINE
    env_file:
      - .env

For more info, here's the documentation I followed.

Hopefully, this helps future readers like myself

Angus McRitchie
  • 1
  • 1