Step 1: Dealing with our wrapping Java code base
For our code, we do:
mvn dependency:tree | grep log4j
And we found some dep from other teams bringing in transitive log4j 1.17. Informed that team and they fixed that in a recent version, we just change in our pom the version to be it, and our pom is fixed.
If your dependency is not maintained anymore, you can enter the artifactory of your organization, and manually look for the classes below in all jars you need(the list is long, because apart from CVE-2021-4104 which mentions JMSAppender, I found log4j 1.x has a lot of other vulnerabilities and more classes should be removed)
org/apache/log4j/net/SocketServer.classorg/apache/log4j/net/SimpleSocketServer.class (just in case)org/apache/log4j/net/SocketAppender.classorg/apache/log4j/net/SMTPAppender$1.classorg/apache/log4j/net/SMTPAppender.classorg/apache/log4j/net/JMSAppender.classorg/apache/log4j/net/JMSSink.classorg/apache/log4j/net/JDBCAppender.classorg/apache/log4j/chainsaw/*.class
If you cannot fix your internal Nexus repo/artifactory, you can find the jar of log4j in local Maven registry (under ~/.m2) and remove the class; then you build your app again; but remember don't use -U to redownload the jar from remote registry.
Step 2: Dealing with base image jars
To find other libs in the base image containing log4j is more complicated.
Tampering the layers by removing the classes files cannot go undetected by Docker daemon. The sha256 value changes, you have to replace the sha256 value in the json file in the main dir with new sha256sum layer.tar; but even with that, Docker daemon will give error when you load the tar: Cannot open /var/lib/docker/tmp-xxxx/...: file not found or so.
Then I tried to create a script to remove the classes at runtime, right before running the app, and define a new entrypoint in jib to run it before running the app.
#!/bin/sh
/opt/amq/bin/fix_log4j_1.x_cves.sh
/opt/amq/bin/launch.sh # the original, inherited entrypoint in jib
But then I found it will slow down pod startup; unresponsive pods may be restarted by Openshift, causing unwanted delay and errors. But the output of this script gives me an idea of which jars contain the classes to remove, which is a solid basis for my next solution.
At last, I came up with a perfect solution
- Implement the previous solution,
docker run the image, and note down the jars' names in the output of the script.
Starting to fix all CVEs regarding Log4j 1.x...
>>>>> Removing class file from '/opt/amq/lib/optional/log4j-1.2.17.redhat-1.jar':
removed 'org/apache/log4j/chainsaw/ControlPanel$1.class'
removed 'org/apache/log4j/chainsaw/ControlPanel$2.class'
...
>>>>> Removing class file from '/opt/amq/activemq-all-5.11.0.redhat-630495.jar':
...
- Define these two as
provided dependencies in pom.xml
- Use maven copy-dependency plugin to copy them into build folder (by default
target/dependency)
- Use maven exec plugin to run the same script against
target/dependency dir to remove vulnerable classes, while building the image with jib
- Use maven jib plugin to copy the fixed jars into the container, so that they will be on a new layer on top of all previous layer, to shadow/whiteout the unfixed jar (see answer to my other question)
By doing this, we eliminate the vulnerable classes while building the image, pod startup speed is not compromised, and the binary transferred to production image registry is already safe.
An advantage of this approach is we are not limited by available tools provided by the container because the script runs now in our local environment. We can install whatever tool we need and use them in the script. For example, in the original script I defined function extract_remove_repackage to complete a simple task of extracting+remove classes+repackaging, only because zip is not installed in the base image. But in my local machine, this can be done by zip in one line,
You have to make sure to bind 3, 4 and 5 to different maven build phases, so that they happen in such order. I bind 3) to compile, 4) to process-classes and 5) to package.
Implementation details below:
- The script (in my previous solution, put under
src/main/jib/opt/amq/bin, so it could be copied into container. Also you needed the new entrypoint script here in the same folder. Now in this solution, moved to src/main/scripts)
fix_log4j_1.x_cves.sh:
#!/bin/bash
# Script to fix log4j 1.x CVEs. Initially it is only for CVE-2021-4104, but
# since there are multiple CVEs regarding log4j 1.x, they are all fixed here:
# Class File CVE
# org/apache/log4j/net/SocketAppender.class CVE-2019-17571
# org/apache/log4j/net/SocketServer.class CVE-2019-17571
# org/apache/log4j/net/SMTPAppender$1.class CVE-2020-9488
# org/apache/log4j/net/SMTPAppender.class CVE-2020-9488
# org/apache/log4j/net/JMSAppender.class CVE-2021-4104
# org/apache/log4j/net/JMSSink.class CVE-2022-23302
# org/apache/log4j/net/JDBCAppender.class CVE-2022-23305
# org/apache/log4j/chainsaw/*.class CVE-2022-23307
cves=(
'CVE-2019-17571'
'CVE-2019-17571'
'CVE-2020-9488'
'CVE-2020-9488'
'CVE-2021-4104'
'CVE-2022-23302'
'CVE-2022-23305'
'CVE-2022-23307'
)
size() {
stat -c %s "$1"
}
extract_remove_repackage() {
before=$1
# jar xf -C some_dir only extract to current dir, we have to cd first
jar_dir=$(dirname "$2")
jar_file=$(basename "$2")
temp_dir=$jar_dir/temp
mkdir "$temp_dir"
cp list.txt "$temp_dir"/ && cp "$2" "$temp_dir"/
cd "$temp_dir"
jar xf "$jar_file"
# provide file and dir names to rm with list.txt
xargs rm -rvf < list.txt && rm list.txt "$jar_file"
jar cf "$jar_file" .
mv "$jar_file" ../
# go back and clean up
cd "$before" && rm -rf "$temp_dir"
}
find_vulnerable_jars() {
cd "$root_dir"
jar -tvf "$1" | grep -E "$pattern" | awk '{ print $8 }' > list.txt
if [ "$(size list.txt)" -gt 0 ]; then
echo ">>>>> Removing class file from '$(realpath "$1")'":
extract_remove_repackage "$(pwd)" "$1"
else
return 0
fi
}
remove_classes_from_jars() {
echo Starting to fix all CVEs regarding Log4j 1.x...
# exclude jolokia.jar(link)
# xargs can return error level to "if", when any of execution fails, while "find -exec" cannot
# because we use custom function, xargs needs "bash -c"; thus we have to use "_" to pass each arg
if find "$root_dir" -name "*.jar" -not -type l -print0 | xargs -0 -n1 bash -c 'find_vulnerable_jars "$@"' _; then
echo All vunerable classes removed. CVE addressed:
printf '%s\n' "${cves[@]}"
else
echo "Error while removing classes; exiting..."
return 1
fi
}
# to be able to use in find -exec child shell, we need to export all vars and functions
# $1: where to search jars, should match copy-dependency output dir.
export root_dir=$1
export pattern=".*(JMS|JDBC|SMTP|Socket)Appender.*.class|.*SocketServer.class|.*JMSSink.class|org/apache/log4j/chainsaw/.*"
export -f size
export -f extract_remove_repackage
export -f find_vulnerable_jars
remove_classes_from_jars
- Define provided dependencies:
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-all</artifactId>
<version>${version.activemq-all}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>${version.log4j}</version>
<scope>provided</scope>
</dependency>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy-cve-jars</id>
<phase>compile</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<includeArtifactIds>activemq-all,log4j</includeArtifactIds>
<includeScope>provided</includeScope>
<includeTypes>jar</includeTypes>
<outputDirectory>${project.build.directory}/dependency</outputDirectory> <!-- default value -->
<excludeTransitive>true</excludeTransitive>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>exec-maven-plugin</artifactId>
<groupId>org.codehaus.mojo</groupId>
<executions>
<execution>
<id>remove-cve-classes</id>
<phase>process-classes</phase>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>${project.build.scriptSourceDirectory}/log4j_cve_fix.sh</executable>
<arguments>
<!-- should match copy-dependency output dir -->
<argument>${project.build.directory}/dependency</argument>
</arguments>
</configuration>
</plugin>
jib plugin: (needs to be > 3.0.0 to be able to use <path><inclueds>)
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<configuration>
<from>
<image>${docker.base.image}</image>
</from>
<to>
<image>${docker.image}</image>
<tags>
<tag>${project.version}</tag>
</tags>
</to>
<container>
<appRoot>/dev/null</appRoot>
<entrypoint>INHERIT</entrypoint> <!-- customized entrypoint not needed anymore, just revert to the way it was -->
</container>
<containerizingMode>packaged</containerizingMode>
<extraDirectories>
<paths>
<path>${project.basedir}/src/main/jib</path>
<path>${project.build.directory}/jib</path>
<path>
<from>target/dependency</from>
<into>/opt/amq/lib/optional</into>
<includes>log4j-${version.log4j}.jar</includes>
</path>
<path>
<from>target/dependency</from>
<into>/opt/amq</into>
<includes>activemq-all-${version.activemq-all}.jar</includes>
</path>
</paths>
<permissions>
<permission>
<!-- don't forget to restrict writing to prevent tampering -->
<file>/opt/amq/conf/log4j.properties</file>
<mode>444</mode>
</permission>
<!-- the copied jars need to be executable -->
<permission>
<file>/opt/amq/lib/${application.executable}</file>
<mode>755</mode>
</permission>
<permission>
<file>/opt/amq/activemq-all-${version.activemq-all}.jar</file>
<mode>755</mode>
</permission>
<permission>
<file>/opt/amq/lib/optional/log4j-${version.log4j}.jar</file>
<mode>755</mode>
</permission>
</permissions>
</extraDirectories>
</configuration>
<executions>
<execution>
<id>jib-build</id>
<phase>package</phase>
<goals>
<goal>${jib.goal}</goal>
</goals>
</execution>
</executions>
</plugin>
