Helpme to get log4j2 correctly

Question

I am newbie to implement log4j2 with my aws java lambda cloud watch. I need custom log instead of cloud watch logs. I am uploading a csv of large size record using step function.So the built in cloud watch logs the same thing repeatedly. So I am planning to add log4j2 with my java lambda. For this I added below dependency in my pom.xml

    <dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-lambda-java-log4j2</artifactId>
    <version>1.0.0</version>
  </dependency>
  <dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.8.2</version>
  </dependency>
  <dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>2.8.2</version>
  </dependency>

Then added the log4j2.xml under src/main/resources. The log4j2.xml is like below

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="com.amazonaws.services.lambda.runtime.log4j2.LambdaAppender">
  <Appenders>
    <Lambda name="Lambda">
      <PatternLayout>
          <pattern>%d{yyyy-MM-dd HH:mm:ss} %X{AWSRequestId} %-5p %c{1}:%L - %m%n</pattern>
      </PatternLayout>
    </Lambda>
  </Appenders>
  <Loggers>
    <Root level="debug">
      <AppenderRef ref="Lambda" />
    </Root>
  </Loggers>
</Configuration>

After that to check the log i have created one aws java lambda project and my code looks like below.

 package com.amazonaws.lambda.demo;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.S3Object;

import com.amazonaws.services.stepfunctions.AWSStepFunctions;
import com.amazonaws.services.stepfunctions.AWSStepFunctionsClientBuilder;
import com.amazonaws.services.stepfunctions.model.StartExecutionRequest;

import com.amazonaws.services.lambda.runtime.Context;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class log4jTest implements RequestHandler<S3Event, String> {
    static final Logger logger = LogManager.getLogger(log4jTest.class); 
    private AmazonS3 s3 = AmazonS3ClientBuilder.standard().build();      
    public log4jTest() {}
    log4jTest(AmazonS3 s3) {
    this.s3 = s3;
    }
@Override
public String handleRequest(S3Event event, Context context) {
    String bucket = event.getRecords().get(0).getS3().getBucket().getName();
    String key = event.getRecords().get(0).getS3().getObject().getKey();
    try {
            for(int i=0;i<10;i++)
                {
                    if(i==10)
                        {
                            logger.error("log data Error");
                        }
                }

        } catch (Exception e) {

                            context.getLogger().log(String.format(
                                "Error getting object %s from bucket %s. Make sure they exist and"
                                    + " your bucket is in the same region as this function.", key, bucket));
                            }
                return null;
}
}

According to the aws document Logging (Java). Everything I did as the doc says. But when I am run the lambda I am getting an error like below

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.

I am checking the log in lambda -> Monitoring -> jump to logs [Screen shot given below]

I googled and went through below sites.

No log4j2 configuration file found. Using default configuration: logging only errors to the console

But I dont know I can't fix this error anyway. Please anyone can help me on this. It would be really grateful if u can do this.

Thanks in advance

...so did you add `-Dorg.apache.logging.log4j.simplelog.StatusLogger.level=TRACE` to see what Log4j2 was doing when it wasn't picking up your log file? — Makoto, Mar 29 '18 at 16:04
Sorry for the late reply.. No I did not add this. Where should I add this? — ShaiNe Ram, Apr 02 '18 at 04:41
Thanks it is solved. I have added the log4j2.xml in the path where the my main lambda class is. Thannk you @Makoto for your reply. — ShaiNe Ram, Apr 03 '18 at 13:35

score 5 · Accepted Answer · answered Jun 28 '18 at 06:55

I found I did not have any luck placing my log4j2.xml file at either src/main/resources or at src/main/resources/path/to/my/Lambda/log4j2.xml. So I did a little digging. In my case, Lambda deploys my handler with a classpath whose first entry is a folder /var/task. The class files are rooted directly at /var/task - but the resource files are placed in /var/task/resources - so Log4J2 cannot find log4j2.xml in the classpath. The solution for me was to use the log4j.configurationFile property to specify the location of the file within the resources folder:

static {
    // System.setProperty("org.apache.logging.log4j.simplelog.StatusLogger.level","TRACE");
    System.setProperty("java.util.logging.manager", "org.apache.logging.log4j.jul.LogManager");
    System.setProperty("log4j.configurationFile", "resources/lambda-log4j2.xml");
}

I also renamed the file "lambda-log4j2.xml" - to make sure it was reading my file, during debugging this... This static block is at the top of my handler class.

The only thing necessary for people with src/main/resources/log4j2.xml is `static { System.setProperty("log4j.configurationFile", "resources/log4j2.xml"); }` — pba, Nov 13 '18 at 02:23

score 0 · Answer 2 · answered Aug 23 '23 at 15:03

This issue is misleading on what the actual issue is, The real issue at play with cloudwatch logs not being generated is an artifact of a known issue where log4j2 does not work correctly with shadowJar or a shaded jar. If you wish to correct this the necessary edits are not related to the log4j2.xml or to your code but to how the application is being built. In order to resolve the issues between log4j2 and shadowJar/shaded jar you will need to apply an additional transformation at build time in order for correct compilation.

Using Gradle

import com.github.jengelman.gradle.plugins.shadow.transformers.Log4j2PluginsCacheFileTransformer

shadowJar{
    transform(Log4j2PluginsCacheFileTransformer)
}

Using Maven

<dependencies>
  ...
  <dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-lambda-java-log4j2</artifactId>
    <version>1.5.1</version>
  </dependency>
  <dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.17.1</version>
  </dependency>
  <dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>2.17.1</version>
  </dependency>
  ....
</dependencies>

and add the following transformation

<plugins>
  ...
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.4.3</version>
    <executions>
      <execution>
        <phase>package</phase>
        <goals>
          <goal>shade</goal>
        </goals>
        <configuration>
          <transformers>
            <transformer
                    implementation="com.github.edwgiz.mavenShadePlugin.log4j2CacheTransformer.PluginsCacheFileTransformer">
            </transformer>
          </transformers>
        </configuration>
      </execution>
    </executions>
    <dependencies>
      <dependency>
        <groupId>com.github.edwgiz</groupId>
        <artifactId>maven-shade-plugin.log4j2-cachefile-transformer</artifactId>
        <version>2.8.1</version>
      </dependency>
    </dependencies>
  </plugin>
  ...
</plugins>

To try to more quickly remediate this issue I have linked all the posts I used to correct this below.

log4j JsonTemplateLayout doesn't work when jar file built by shadowJar Original post that clued me in
Console contains an invalid element or attribute "JsonTemplateLayout" even after adding dependency Full example of how to fix in maven or gradle
How correctly log AWS Lambda event with lambda powertools Explanation with powertools
Logging using Log4J2 on aws lambda - Class not found Explanation where someone is claiming the issue is due to a bad path name (that is probably not the case)

Helpme to get log4j2 correctly

2 Answers2