azure-spring-boot-starter-keyvault-secrets not resolving vcap service variables during application startup in PCF

Question

I am upgrading my application from older version of azure-key-vault to latest version of azure bom.

Below are the changes in pom.xml and the app fails to start in PCF

Before making changes in pom.xml:

    <dependency>
      <groupId>com.microsoft.azure</groupId>
      <artifactId>azure-keyvault</artifactId>
      <version>1.2.4</version>
    </dependency>
    <dependency>
      <groupId>com.microsoft.azure</groupId>
      <artifactId>azure-keyvault-secrets-spring-boot-starter</artifactId>
      <scope>runtime</scope>
      <version>2.1.3</version>
    </dependency>
    <dependency>
      <groupId>com.microsoft.azure</groupId>
      <artifactId>azure-storage</artifactId>
      <version>5.5.0</version>
    </dependency>

After making changes in pom.xml


    <dependency>
      <groupId>com.azure.spring</groupId>
      <artifactId>azure-spring-boot-starter-keyvault-secrets</artifactId>
    </dependency>
    .....
    <dependencyManagement>
         <dependency>
            <groupId>com.azure.spring</groupId>
            <artifactId>azure-spring-boot-bom</artifactId>
            <version>3.5.0</version>
            <type>pom</type>
            <scope>import</scope>
          </dependency>
        </dependencies>
      </dependencyManagement>

When using older version, the app starts and was able to read the

azure.keyvault.enabled=true
azure.keyvault.client-id=${vcap.services.myservice.credentials.clientId}
azure.keyvault.client-key=${vcap.services.myservice.credentials.clientKey}
azure.keyvault.uri=${vcap.services.myservice.credentials.uri}
azure.keyvault.tenant-id=1234567-1234-1234-1234-1234werewrew

property on application startup

But the same is not getting resolved after making the pom.xml changes and getting the below exception

2021-07-22T00:15:38.945-07:00 [CELL/0] [OUT] Downloaded droplet (167.3M)
2021-07-22T00:15:42.169-07:00 [APP/PROC/WEB/0] [OUT] 2021-07-22 07:15:42.166 INFO 15 --- [ main] pertySourceApplicationContextInitializer : 'cloud' property source added
2021-07-22T00:15:42.170-07:00 [APP/PROC/WEB/0] [OUT] 2021-07-22 07:15:42.170 INFO 15 --- [ main] nfigurationApplicationContextInitializer : Reconfiguration enabled
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] 2021-07-22 07:15:51.846 ERROR 15 --- [ main] o.s.boot.SpringApplication : Application run failed
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] java.lang.IllegalStateException: Failed to configure KeyVault property source
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:121) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:47) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:100) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:86) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:82) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:63) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:117) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:111) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:62) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:375) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.run(SpringApplication.java:333) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.TestApplication.main(TestApplication.java:23) [classes/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] Caused by: java.lang.RuntimeException: Max retries 3 times exceeded. Error Details: ManagedIdentityCredential authentication unavailable. Connection to IMDS endpoint cannot be established, connect timed out.
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.core.http.policy.RetryPolicy.lambda$attemptAsync$1(RetryPolicy.java:127) ~[azure-core-1.16.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onError(FluxPeekFuseable.java:553) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onError(FluxPeekFuseable.java:553) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onError(FluxPeekFuseable.java:553) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.Operators.error(Operators.java:197) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:135) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:199) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:173) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2663) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onComplete(MonoFlatMapMany.java:260) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:150) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:150) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:145) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxHandle$HandleSubscriber.onComplete(FluxHandle.java:212) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:269) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:401) ~[reactor-netty-core-1.0.8.jar:1.0.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:420) ~[reactor-netty-core-1.0.8.jar:1.0.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:474) ~[reactor-netty-core-1.0.8.jar:1.0.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:685) ~[reactor-netty-http-1.0.8.jar:1.0.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94) ~[reactor-netty-core-1.0.8.jar:1.0.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1368) ~[netty-handler-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1245) ~[netty-handler-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282) ~[netty-handler-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[netty-transport-native-epoll-4.1.65.Final-linux-x86_64.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-native-epoll-4.1.65.Final-linux-x86_64.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-native-epoll-4.1.65.Final-linux-x86_64.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] Suppressed: java.lang.Exception: #block terminated with an error
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.Flux.blockLast(Flux.java:2519) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:94) ~[azure-core-1.16.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.core.util.paging.ContinuablePagedByPageIterable$ContinuablePagedByPageIterator.<init>(ContinuablePagedByPageIterable.java:49) ~[azure-core-1.16.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.core.util.paging.ContinuablePagedByPageIterable.iterator(ContinuablePagedByPageIterable.java:37) ~[azure-core-1.16.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.lang.Iterable.spliterator(Iterable.java:101) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultOperation.lambda$refreshProperties$2(KeyVaultOperation.java:142) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.util.Optional.map(Optional.java:215) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultOperation.refreshProperties(KeyVaultOperation.java:142) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultOperation.<init>(KeyVaultOperation.java:80) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:101) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:47) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:100) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:86) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:82) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:63) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:117) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:111) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:62) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:375) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.run(SpringApplication.java:333) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.TestApplication.main(TestApplication.java:23) [classes/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] Caused by: com.azure.identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. Connection to IMDS endpoint cannot be established, connect timed out.
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.identity.implementation.IdentityClient.lambda$checkIMDSAvailable$50(IdentityClient.java:1208) ~[azure-identity-1.3.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoCallable.call(MonoCallable.java:91) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:127) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] ... 55 common frames omitted
2021-07-22T00:15:51.848-07:00 [APP/PROC/WEB/0] [OUT] Caused by: java.net.SocketTimeoutException: connect timed out
2021-07-22T00:15:51.848-07:00 [APP/PROC/WEB/0] [OUT] at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_272]
2021-07-22T00:15:51.848-07:00 [APP/PROC/WEB/0] [OUT] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_272]
2021-07-22T00:15:51.848-07:00 [APP/PROC/WEB/0] [OUT] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_272]
....
2021-07-22T00:15:51.848-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.identity.implementation.IdentityClient.lambda$checkIMDSAvailable$50(IdentityClient.java:1202) ~[azure-identity-1.3.0.jar:na]
2021-07-22T00:15:52.202-07:00 [APP/PROC/WEB/0] [OUT] Exit status 1
2021-07-22T00:15:52.204-07:00 [CELL/SSHD/0] [OUT] Exit status 0

Has anything changed which is breaking this?

I would suggest try azure-keyvault latest version 3.0.5 to cope with latest version of azure-spring-boot-bom 3.5.0 Reference - https://www.nuget.org/packages/Microsoft.Azure.KeyVault — RahulKumarShaw, Jul 26 '21 at 11:23
Have it only web dependency: com.azure.spring.boot azure-spring-boot-starter-web Instead of, com.azure.spring azure-spring-boot-starter-keyvault-secrets Reference : https://stackoverflow.com/questions/54458324/o-s-boot-springapplication-application-run-failed — RahulKumarShaw, Jul 26 '21 at 11:24
@RahulKumarShaw-MT Thanks for the comments, but the mentioned artifacts are deprecated and I need to use the latest artifacts — Venu, Jul 26 '21 at 21:28
The error is ```Caused by: com.azure.identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. Connection to IMDS endpoint cannot be established, connect timed out.``` — Venu, Jul 26 '21 at 22:10
If I pass the same vcap variables as environment properties in my pcf directly, the app starts and works without issues. — Venu, Jul 29 '21 at 00:19

chenrujun · Answer 1 · 2021-07-30T06:57:46.830

0

@VenuMadhav.

I tried to use azure-spring-boot-starter-keyvault-secrets:3.5.0 and spring-boot:2.4.8, there is no problem. So I can not reproduce your problem.

If I pass the same vcap variables as environment properties in my pcf directly, the app starts and works without issues.

Do you mean when you put vcap variables in Key Vault, then the app can not start?

If your problem still exists, could you please create a simple sample in github which can reproduce you problem, and share the link of the sample?

And it's better to ask questions by github issues: https://github.com/Azure/azure-sdk-for-java/issues

edited Jul 30 '21 at 06:57

answered Jul 30 '21 at 06:32

chenrujun

126
4

Hi, @VenuMadhav, thank you for your quick response. I can not reproduce your problem. Could you please create a sample project to reproduce your problem, and share it by github? – chenrujun Jul 30 '21 at 06:56
Thanks for trying. Below is my use case I've azure-key-vault related details like client id, client-key, tenant-id as part of vcap_services i.e. pcf variables and not directly set in application.properties or passed via pcf environment variables. So, when the app tries to start with the above updated pom.xml, it fails. But if I add those vales in application.properties, the app starts successfully. – Venu Jul 30 '21 at 06:59
1. https://github.com/Azure/azure-sdk-for-java/blob/azure-spring-boot-starter-active-directory_3.7.0/sdk/spring/azure-spring-boot/src/main/java/com/azure/spring/keyvault/KeyVaultEnvironmentPostProcessor.java#L17 2. https://github.com/Azure/azure-sdk-for-java/blob/azure-spring-boot-starter-active-directory_3.7.0/sdk/spring/azure-spring-cloud-autoconfigure/src/main/java/com/azure/spring/cloud/autoconfigure/cloudfoundry/AzureCloudFoundryEnvironmentPostProcessor.java#L38 Could you please try to add dependency of `azure-spring-cloud-autoconfigure`? – chenrujun Jul 30 '21 at 07:57

Venu · Accepted Answer · 2021-08-06T07:17:28.153

Finally I was able to resolve the issue

Few properties are defined with camel case as below clientId and clientKey in my application.properties file and those are causing the issue.

azure.keyvault.client-id=${vcap.services.myservice.credentials.clientId}
azure.keyvault.client-key=${vcap.services.myservice.credentials.clientKey}

After changing the place holders from camel case to lower case, the app was able to resolve those properties. I changed the place holders as below

azure.keyvault.client-id=${vcap.services.myservice.credentials.clientid}
azure.keyvault.client-key=${vcap.services.myservice.credentials.clientkey}

NOTE: I did not change to lower case in PCF service bindings and they are still with camel case. So, seems like spring-boot is converting all the keys to lower case

azure-spring-boot-starter-keyvault-secrets not resolving vcap service variables during application startup in PCF

2 Answers2