I am currently running Scala Spark applications on EMR serverless and all of the logs are getting output to stderr and logged at info level. Looking at this page it seems like this is the default for spark;
https://github.com/apache/spark/blob/v3.3.0/conf/log4j2.properties.template
I have been able to adjust the logging level using these lines based on the info here https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/log4j2.html
"rootLogger.level": "warn"
"logger.myapp.name": "com.myapp",
"logger.myapp.level": "info",
but everything is still coming out to stderr. How would I configure this to have my application logs appear in stdout instead? I've tried the below configuration but it doesn't have any effect.
{
"classification": "spark-executor-log4j2",
"configurations": [],
"properties": {
"rootLogger.level": "warn",
"appender.myapp.target": "stdout",
"appender.myapp.layout.type": "PatternLayout",
"appender.myapp.name": "myapp",
"appender.myapp.type": "Console",
"logger.myapp.name": "com.myapp",
"logger.myapp.level": "info",
"logger.myapp.appenderRef.myappout.ref": "myapp",
"logger.myapp.appenderRefs": "myappout"
}
}