Jenkins Startup fails on AWS ECS due to secrets-manager-credentials-provider-plugin

Question

I'm using the AWS Secrets Manager Credentials Provider plugin and it seems to be causing Jenkins to fail on startup.

I've followed the troubleshooting steps here with no luck, the last thing I did was split out the IAM perms.

I am running the Jenkins/Jenkins:lts docker image, on AWS ECS, and describing my stack using AWS CDK. i installed the plugin using /usr/local/bin/install-plugins.sh from the docker image.

When I run the same docker image on an EC2 server startup is succesful but via ECS i get this error.

java.lang.NullPointerException
    at io.jenkins.plugins.credentials.secretsmanager.AwsSecretSource.reveal(AwsSecretSource.java:35)
    at io.jenkins.plugins.casc.SecretSourceResolver$ConfigurationContextStringLookup.lambda$lookup$ad236547$1(SecretSourceResolver.java:141)
    at io.vavr.CheckedFunction0.lambda$unchecked$52349c75$1(CheckedFunction0.java:247)
    at io.jenkins.plugins.casc.SecretSourceResolver$ConfigurationContextStringLookup.lambda$lookup$0(SecretSourceResolver.java:141)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
    at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632)
    at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
    at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:543)
    at io.jenkins.plugins.casc.SecretSourceResolver$ConfigurationContextStringLookup.lookup(SecretSourceResolver.java:143)
    at org.apache.commons.text.lookup.InterpolatorStringLookup.lookup(InterpolatorStringLookup.java:144)
    at org.apache.commons.text.StringSubstitutor.resolveVariable(StringSubstitutor.java:1067)
    at org.apache.commons.text.StringSubstitutor.substitute(StringSubstitutor.java:1433)
    at org.apache.commons.text.StringSubstitutor.substitute(StringSubstitutor.java:1308)
    at org.apache.commons.text.StringSubstitutor.replaceIn(StringSubstitutor.java:1019)
    at io.jenkins.plugins.casc.SecretSourceResolver.resolve(SecretSourceResolver.java:109)
    at io.jenkins.plugins.casc.impl.configurators.PrimitiveConfigurator.configure(PrimitiveConfigurator.java:44)
    at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.tryConstructor(DataBoundConfigurator.java:159)
    at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.instance(DataBoundConfigurator.java:76)
    at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:267)
    at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.configure(DataBoundConfigurator.java:82)
    at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$doConfigure$16668e2$1(HeteroDescribableConfigurator.java:277)
    at io.vavr.CheckedFunction0.lambda$unchecked$52349c75$1(CheckedFunction0.java:247)
    at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.doConfigure(HeteroDescribableConfigurator.java:277)
    at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$configure$2(HeteroDescribableConfigurator.java:86)
    at io.vavr.control.Option.map(Option.java:392)
    at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$configure$3(HeteroDescribableConfigurator.java:86)
    at io.vavr.Tuple2.apply(Tuple2.java:238)
    at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:83)
    at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:55)
    at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.tryConstructor(DataBoundConfigurator.java:151)
    at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.instance(DataBoundConfigurator.java:76)
    at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:267)
    at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.check(DataBoundConfigurator.java:100)
    at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:344)
    at io.jenkins.plugins.casc.BaseConfigurator.check(BaseConfigurator.java:287)
    at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:351)
    at io.jenkins.plugins.casc.BaseConfigurator.check(BaseConfigurator.java:287)
    at io.jenkins.plugins.casc.ConfigurationAsCode.lambda$checkWith$8(ConfigurationAsCode.java:777)
    at io.jenkins.plugins.casc.ConfigurationAsCode.invokeWith(ConfigurationAsCode.java:713)
    at io.jenkins.plugins.casc.ConfigurationAsCode.checkWith(ConfigurationAsCode.java:777)
    at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:762)
    at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:638)
    at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:307)
    at io.jenkins.plugins.casc.ConfigurationAsCode.init(ConfigurationAsCode.java:299)
Caused: java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
Caused: java.lang.Error
    at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
    at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1129)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused: org.jvnet.hudson.reactor.ReactorException
    at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
    at jenkins.InitReactorRunner.run(InitReactorRunner.java:49)
    at jenkins.model.Jenkins.executeReactor(Jenkins.java:1162)
    at jenkins.model.Jenkins.<init>(Jenkins.java:960)
    at hudson.model.Hudson.<init>(Hudson.java:86)
    at hudson.model.Hudson.<init>(Hudson.java:82)
    at hudson.WebAppMain$3.run(WebAppMain.java:295)
Caused: hudson.util.HudsonFailedToLoad
    at hudson.WebAppMain$3.run(WebAppMain.java:312)

I have also submitted an issue. https://github.com/jenkinsci/aws-secrets-manager-credentials-provider-plugin/issues/117#issue-938932773

Koroslak · Answer 1 · 2022-01-28T08:32:45.533

EDIT: In the like from your question is a simpler solution: To use AWS_REGION. I was using AWS_DEFAULT_REGION which isn't working.

Here's my simplified solution.

# install aws cli
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_REGION
ENV AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    AWS_REGION=${AWS_REGION}
RUN wget "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -O "awscliv2.zip" \
    && unzip -q awscliv2.zip \
    && ./aws/install \
    && rm awscliv2.zip

for completeness here's my make build command

build:
    echo -e $(shell getent group docker | cut -d: -f3)
    docker build -t $(PROJ):$(VERSION) --build-arg DOCKER_GID=$(shell getent group docker | cut -d: -f3) \
    --build-arg AWS_ACCESS_KEY_ID=$(shell aws configure get aws_access_key_id --profile=default) \
    --build-arg AWS_SECRET_ACCESS_KEY=$(shell aws configure get aws_secret_access_key --profile=default) \
    --build-arg AWS_REGION=$(shell aws configure get region --profile=default) \
    -f Dockerfile .

OLD ANSWER:

Looks like Hudson uses credentials and config from ~/.aws Problem with Jenkins and docker is that $JENKINS_HOME is a volume (as I learned here), not a folder. You can't create folders in there from Dockerfile.

Luckily you can change the location of those configs.

Here's part of my Dockerfile that solved the issue:

ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_DEFAULT_REGION
ENV AWS_CONFIG_FOLDER=/opt/.aws
ENV AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} \
    AWS_CONFIG_FILE=$AWS_CONFIG_FOLDER/config \
    AWS_SHARED_CREDENTIALS_FILE=$AWS_CONFIG_FOLDER/credentials
RUN wget "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -O "awscliv2.zip" \
    && unzip -q awscliv2.zip \
    && ./aws/install \
    && rm awscliv2.zip \
    && mkdir $AWS_CONFIG_FOLDER \
    && aws --profile default configure set aws_access_key_id "$AWS_ACCESS_KEY_ID" \
    && aws --profile default configure set aws_secret_access_key "$AWS_SECRET_ACCESS_KEY" \
    && aws --profile default configure set region "$AWS_DEFAULT_REGION" \
    && chown -R jenkins:jenkins $AWS_CONFIG_FOLDER

Adding AWS credentials to the container image (`ENV`) is a serious security issue. Please don't do that, especially if you publish those container images. — Nils Ballmann, Mar 09 '22 at 09:18

Jenkins Startup fails on AWS ECS due to secrets-manager-credentials-provider-plugin

1 Answers1