CloudHub Notification - Fails connections from workers in VPC

Question

I developed an API which will trigger email alerts based on some business requirements. To raise email alerts, I used cloudHub create-notification connector. The connector configuration is below:

<cloudhub:config name="CloudHub_Config" doc:name="CloudHub Config">
    <cloudhub:connection 
        username="${ch.username}" 
        password="${ch.password}" 
        environment="${environment.id}"/>
</cloudhub:config>

Username and password are configured in Runtime manager > Application > properties, and the user has required permissions which I verified in Access Management. A Custom Application Notification alert is configured on correct environment in Runtime manager.

I am using below code to generate alerts from mule flows.

<cloudhub:create-notification doc:name="Success" config-ref="CloudHub_Config" domain="${domain}" priority="INFO">
    <cloudhub:message><![CDATA[#["
        Success scenario 
        Hello user"]]]>
    </cloudhub:message>
    <cloudhub:custom-properties><![CDATA[#[output application/java
        ---
        {
            "category" : "Info",
            "status" : "Success"
        }]]]>
    </cloudhub:custom-properties>
</cloudhub:create-notification>

I tested this flow on 4.4.0 (latest) runtime on Studio pointing to AnyPoint platform TEST environment, and it was success. I deployed the same code to cloudHub worker (within VPC) and it failed test-connection during application start-up. The deployment went through and started successfully, but it fails when executing the component from flows. The error details is:

Message               : Unexpected Error Occurred
Error type            : MULE:UNKNOWN
Payload Type          : org.mule.runtime.core.internal.streaming.bytes.ManagedCursorStreamProvider
Root Exception stack trace:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:445)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:317)
    at sun.security.validator.Validator.validate(Validator.java:262)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:289)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1675)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:226)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:1015)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:1012)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1504)
    at org.glassfish.grizzly.ssl.SSLUtils.executeDelegatedTask(SSLUtils.java:274)
    at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:735)
    at org.glassfish.grizzly.ssl.SSLFilter.doHandshakeStep(SSLFilter.java:345)
    at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:648)
    at org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:349)
    at com.ning.http.client.providers.grizzly.SwitchingSSLFilter.handleRead(SwitchingSSLFilter.java:74)
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:540)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
    at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
    at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415)
    at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384)
    at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348)
    at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:151)
    at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(RunnableFutureDecorator.java:54)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

CloudHub Notification component looks like not able to connect to platform APIs. The below logs shows the error while start-up of application:

08:45:46.011     07/26/2022     Worker-0     [MuleRuntime].uber.02: [myCompany-api-v1].uber@org.mule.runtime.module.extension.internal.runtime.config.LifecycleAwareConfigurationInstance.testConnectivity:189 @5176f3d4     INFO
Connectivity test failed for config 'CloudHub_Config'. Application deployment will continue. Error was: Unexpected Error occurred trying to validate the connection.
java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at com.mulesoft.connectors.cloudhub.internal.CloudHubConnectionProvider.validate(CloudHubConnectionProvider.java:111)
    at com.mulesoft.connectors.cloudhub.internal.CloudHubConnectionProvider.validate(CloudHubConnectionProvider.java:42)
    at org.mule.runtime.module.extension.internal.runtime.config.ClassLoaderConnectionProviderWrapper.validate(ClassLoaderConnectionProviderWrapper.java:72)
    at org.mule.runtime.core.internal.connection.ReconnectableConnectionProviderWrapper.validate(ReconnectableConnectionProviderWrapper.java:50)
    at org.mule.runtime.core.internal.connection.ErrorTypeHandlerConnectionProviderWrapper.validate(ErrorTypeHandlerConnectionProviderWrapper.java:85)
    at org.mule.runtime.core.internal.connection.DefaultConnectionManager.doTestConnectivity(DefaultConnectionManager.java:186)
    at org.mule.runtime.core.internal.connection.DefaultConnectionManager.lambda$testConnectivity$1(DefaultConnectionManager.java:169)
    at org.mule.runtime.core.internal.connection.DefaultConnectionManager.doTestConnectivity(DefaultConnectionManager.java:176)
    at org.mule.runtime.core.internal.connection.DefaultConnectionManager.testConnectivity(DefaultConnectionManager.java:152)
    at org.mule.runtime.core.internal.connection.DelegateConnectionManagerAdapter$EagerConnectionManagerAdapter.testConnectivity(DelegateConnectionManagerAdapter.java:176)
    at org.mule.runtime.core.internal.connection.DelegateConnectionManagerAdapter.testConnectivity(DelegateConnectionManagerAdapter.java:98)
    at org.mule.runtime.module.extension.internal.runtime.config.LifecycleAwareConfigurationInstance$1.doWork(LifecycleAwareConfigurationInstance.java:204)
    at org.mule.runtime.core.api.retry.policy.AbstractPolicyTemplate.execute(AbstractPolicyTemplate.java:62)
    at org.mule.runtime.core.internal.retry.async.RetryWorker.run(RetryWorker.java:56)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:152)
    at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(RunnableFutureDecorator.java:54)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
    at com.mulesoft.connectors.cloudhub.internal.CloudHubConnectionProvider.validate(CloudHubConnectionProvider.java:93)
    ... 20 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1566)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:545)
    at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1217)
    at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1185)
    at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:471)
    at org.glassfish.grizzly.ssl.SSLUtils.sslEngineWrap(SSLUtils.java:451)
    at org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:360)
    at org.glassfish.grizzly.ssl.SSLUtils.handshakeWrap(SSLUtils.java:327)
    at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:724)
    at org.glassfish.grizzly.ssl.SSLFilter.doHandshakeStep(SSLFilter.java:345)
    at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:648)
    at org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:349)
    at com.ning.http.client.providers.grizzly.SwitchingSSLFilter.handleRead(SwitchingSSLFilter.java:74)
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:540)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
    at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
    at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415)
    at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384)
    at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348)
    at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279)
    ... 7 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:198)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1729)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:333)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:325)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1688)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:226)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:1015)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:1012)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1504)
    at org.glassfish.grizzly.ssl.SSLUtils.executeDelegatedTask(SSLUtils.java:274)
    at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:735)
    ... 25 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:450)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:317)
    at sun.security.validator.Validator.validate(Validator.java:262)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:289)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1675)
    ... 33 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:445)
    ... 39 more

Scenarios I tried:

Running the same code on 4.3.0 runtime on same cloudHub worker within VPC - IT FAILED.
Running the same code on 4.4.0 (latest) runtime on another AnyPoint platform's cloudHub worker which is not on VPC - IT WORKED FINE.

Seems like connectivity of cloudHub-notification component running on worker within VPC is causing SSL handshake error. Strange that this component doesn't provide any settings to configure SSL.

Has anyone faced this issue before, if yes - please help me here?

Did you test deploying in the VPC with Mule runtime 4.4.0? Are you setting any Java argument in properties? What's the version of the connector? — aled, Jul 26 '22 at 12:52
Hi @aled, Yes - I did test deploying in the VPC with runtime 4.4.0 and 4.3.0, it doesn't work. I am not setting any java arguments in runtime properties, just setting ch.username and ch.password. CloudHub connector version is v1.0.1. — Sambit Swain, Jul 26 '22 at 14:24

Sambit Swain · Answer 1 · 2023-03-21T21:31:04.310

0

This is an update if anybody else facing this same problem.

Issue: The Trust Store configuration for LDAPS connection is not bound only to the LDAP Connector, rather it impacts other connector secured connections within same API. LDAP connection (i.e., without trust store) will not cause any issue.

MuleSoft Support have now confirmed and validated this as BUG in LDAP connector (all versions), an ER is in-progress by MuleSoft technical team to release fix.

Hopefully new version of LDAP would solve the problem.

Meanwhile suggested workaround is to import additional certificate to LDAP trust store file. That would help both LDAPS connection and other secured connection to work properly.

edited Mar 21 '23 at 21:31

answered Feb 24 '23 at 16:36

Sambit Swain

131
1
13

Is this is a bug in a specific version of a connector or Mule? Mentioning that would help others to identify if they could be impacted by the issue. – aled Feb 24 '23 at 17:19
Yes, it was a bug. The LDAP Connector version 3.5.3 is now available with the fix. – Sambit Swain Apr 07 '23 at 09:17

CloudHub Notification - Fails connections from workers in VPC

1 Answers1