5

I am using an AWS Managed Airflow to orchestrate a data pipeline. The problem that I am facing now is that logs are not being printed in the airflow log not on cloud watch.

I am getting the following error in the log of Air flow:

*** Reading remote log from Cloudwatch log_group: airflow-Task log_stream: 2021-08-26T08_08_24+00_00/1.log.
Could not read remote logs from log_group: airflow-Task log_stream: 2021-08-26T08_08_24+00_00/1.log.
Birat Bade Shrestha
  • 800
  • 1
  • 8
  • 28

4 Answers4

3

AWS has a troubleshooting guide for this specific case.

*** Reading remote log from Cloudwatch log_group: airflow-{environmentName}-Task log_stream: {DAG_ID}/{TASK_ID}/{time}/{n}.log.Could not read remote logs from log_group: airflow-{environmentName}-Task log_stream: {DAG_ID}/{TASK_ID}/{time}/{n}.log.

https://docs.aws.amazon.com/mwaa/latest/userguide/t-cloudwatch-cloudtrail-logs.html#t-task-logs

I would check the following:

  1. Verify that you enabled task logs at the INFO level for your environment.
  2. Check if the MWAA Execution role has CloudWatch read access policy attached.
  3. Eventually try adding apache-airflow[amazon] provider to the MWAA Requirements file.
Jacek Sztandera
  • 904
  • 6
  • 5
1

It was an issue related to role and policies.

MWAA Execution role didn't have the required CloudWatch access policy attached to it.

Birat Bade Shrestha
  • 800
  • 1
  • 8
  • 28
  • can u shed further light on this. we are facing similar issue and are unable to figure it out. – Priyadarshan Mohanty Nov 18 '21 at 17:23
  • 2
    @PriyadarshanMohanty in my case this was an issue: https://github.com/idealo/terraform-aws-mwaa/issues/42 – Anton Bryzgalov Dec 27 '21 at 15:51
  • For me re-deploying terraform from us-east-1 to eu-central-1 also solved the issue. As far as I remember, without KMS encryption logs were available at us-east-1 as well. – VB_ Dec 31 '21 at 14:21
  • Hi @AntonBryzgalov, is this the backend script that's used in MWAA? I think the issue is more related to the worker getting killed before the task gets scheduled on the worker node. – Priyadarshan Mohanty May 31 '22 at 05:28
  • Hi @PriyadarshanMohanty, no, this is just an externally maintained Terraform module for MWAA. Not related to MWAA internals, but due to region mismatch I have had the same issue as in the post. – Anton Bryzgalov Jun 01 '22 at 16:35
  • 2
    What is the required CloudWatch access policy ? – linSESH Oct 05 '22 at 15:11
0

In my case the issue was because of encryption. Check if your MWAA uses the same KMS key as your CloudWatch Log Groups.

If MWAA is created with custom KMS key this key is also used for CloudWatch Log Groups encryption. When you delete and re-create MWAA (with new KMS key) Log Groups are not touched and remain encrypted with old key. And they are not accessible for MWAA any more.

Two ways to fix it:

  1. Delete Log Groups, create new Log Groups with the same name and using KMS key which is used by MWAA.
  2. Don't use KMS key when you create MWAA. In this case, AWS managed key aws/airflow will be used automatically. It should help to avoid dependency on particular KMS key.
Alena Melnikova
  • 931
  • 7
  • 9
0

In My case, this issue happened because the KMS used for encrypting the S3 bucket and MWAA was missing following in the Key policy

"kms:GenerateDataKey*",

for principal

AWS": "arn:aws:iam:::root"

Sameer
  • 1
  • 1
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Oct 21 '22 at 11:28