DestinationAccessException on keystore location for SuccessFactors destination

Question

I am having a problem accessing a SuccessFactors destination in my application. I am trying to access the destination like:

 final HttpDestination destination = DestinationAccessor.getDestination(DESTINATION_NAME).asHttp();

where DESTINATION_NAME is "sfapi_dest". This destination is generated automatically using an instance of the SCP SuccessFactors Extensibility Service (Extension Factory) which creates an OAuth2SAMLBearerAssertion protected destination. The application is secured in the usual way using approuter and xsuaa service. However, when I run my application I get a DestinationAccessException on the keystore location. I am using version 3.10.0 of the SDK.

Here is the destination:

Earlier in the logs I can see:

"level":"DEBUG","categories":[],"msg":"Instantiated com.sap.cloud.sdk.cloudplatform.connectivity.DefaultDestination based on the following property keys: apiKey,audience,Authentication,authnContextClassRef,authTokens,certificates,clientKey,companyId,KeyStoreLocation,KeyStorePassword,Name,nameIdFormat,ProxyType,tokenServiceURL,Type,URL,WebIDEEnabled,XFSystemName" }

and then:

"log","logger":"com.sap.cloud.sdk.cloudplatform.connectivity.DestinationLoaderChain","thread":"http-nio-0.0.0.0-8080-exec-3","level":"DEBUG","categories":[],"msg":"Destination loader ScpCfDestinationLoader successfully loaded destination sfapi_dest." }

and finally:

"msg":"Servlet.service() for servlet [com.sap.cloud.sdk.sfcrud.CandidatesServlet] in context with path [] threw exception","stacktrace":["com.sap.cloud.sdk.cloudplatform.exception.ShouldNotHappenException: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Failed to find key store 'sfapi_dest.p12' in destination 'sfapi_dest'.","\tat com.sap.cloud.sdk.cloudplatform.servlet.RequestAccessorFilter.doFilter(RequestAccessorFilter.java:74)","\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)","\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)","\tat com.sap.cloud.sdk.cloudplatform.security.servlet.HttpCachingHeaderFilter.doFilter(HttpCachingHeaderFilter.java:83)","\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)","\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)","\tat com.sap.cloud.sdk.cloudplatform.security.servlet.HttpSecurityHeadersFilter.doFilter(HttpSecurityHeadersFilter.java:41)","\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)","\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)","\tat org.apache.catalina.filters.RestCsrfPreventionFilter.doFilter(RestCsrfPreventionFilter.java:125)","\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)","\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)","\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)","\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)","\tat org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:44)","\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:668)","\tat com.sap.xs.security.container.XSSecurityAuthenticator.invoke(XSSecurityAuthenticator.java:134)","\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)","\tat org.apache.tomee.catalina.OpenEJBSecurityListener$RequestCapturer.invoke(OpenEJBSecurityListener.java:97)","\tat com.sap.xs.java.valves.ErrorReportValve.invoke(ErrorReportValve.java:66)","\tat ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:256)","\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)","\tat com.sap.xs.security.TenantIdValve.invoke(TenantIdValve.java:33)","\tat com.sap.xs.security.UserInfoValve.invoke(UserInfoValve.java:19)","\tat com.sap.xs.statistics.tomcat.valve.RequestTracingValve.invoke(RequestTracingValve.java:43)","\tat com.sap.xs.logging.catalina.RuntimeInfoValve.invoke(RuntimeInfoValve.java:40)","\tat org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)","\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)","\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:609)","\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)","\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)","\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1623)","\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)","\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)","\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)","\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)","\tat java.lang.Thread.run(Thread.java:836)","Caused by: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Failed to find key store 'sfapi_dest.p12' in destination 'sfapi_dest'.","\tat com.sap.cloud.sdk.cloudplatform.thread.AbstractThreadContextExecutor.execute(AbstractThreadContextExecutor.java:325)","\tat com.sap.cloud.sdk.cloudplatform.servlet.RequestAccessorFilter.doFilter(RequestAccessorFilter.java:71)","\t... 36 more","Caused by: com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Failed to find key store 'sfapi_dest.p12' in destination 'sfapi_dest'.","\tat com.sap.cloud.sdk.cloudplatform.connectivity.ScpCfHttpDestinationPropertyFactory.getKeyStore(ScpCfHttpDestinationPropertyFactory.java:432)","\tat com.sap.cloud.sdk.cloudplatform.connectivity.ScpCfHttpDestination.lambda$new$0(ScpCfHttpDestination.java:146)","\tat io.vavr.control.Option.orElse(Option.java:321)","\tat com.sap.cloud.sdk.cloudplatform.connectivity.ScpCfHttpDestination.(ScpCfHttpDestination.java:145)","\tat com.sap.cloud.sdk.cloudplatform.connectivity.ScpCfHttpDestination.(ScpCfHttpDestination.java:79)","\tat com.sap.cloud.sdk.cloudplatform.connectivity.ScpCfDestination.asHttp(ScpCfDestination.java:49)","\tat com.sap.cloud.sdk.sfcrud.CandidatesServlet.doGet(CandidatesServlet.java:38)","\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:634)","\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:741)","\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)","\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)","\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)","\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)","\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)","\tat org.apache.openejb.server.httpd.EEFilter.doFilter(EEFilter.java:65)","\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)","\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)","\tat com.sap.cloud.sdk.cloudplatform.servlet.RequestAccessorFilter.lambda$doFilter$1(RequestAccessorFilter.java:71)","\tat com.sap.cloud.sdk.cloudplatform.thread.AbstractThreadContextExecutor.lambda$execute$0(AbstractThreadContextExecutor.java:317)","\tat com.sap.cloud.sdk.cloudplatform.thread.ThreadContextCallable.call(ThreadContextCallable.java:247)","\tat com.sap.cloud.sdk.cloudplatform.thread.AbstractThreadContextExecutor.execute(AbstractThreadContextExecutor.java:319)","\t... 37 more"] }

I can see the KeyStoreLocation in "sensitive information" of the instance of the SuccessFactors Extensibility Service but this information does not seem to be added to the automatically generated destination or at least it is not visible. Any idea why it fails to find the keystore location in the DestinationAccessor request? Thanks for your time.

Adding image:

Please attach the whole log entries, attach a screenshot of the destination and tell us which SDK Version you use. — Emdee, May 13 '20 at 17:19
I have added the destination configuration image and am using version 3.18.0 of the SDK. The whole log entry would be too big for here and contain a lot of sensitive information. Is there a particular part you want to see? — Inkers, May 14 '20 at 08:40
I have expanded the error log, which shows the error coming from the getKeyStore() I think — Inkers, May 14 '20 at 09:37
Where is the notepad-screenshot with the destination properties coming from? (a) From your local computer env variable "destinations"? If yes, then where did you get the data from and was there a key file documented somewhere else? (b) From a "destinations" property in (CF) "User-Provided Variables"? If yes, then you should not combine the usage of "destinations" env variable and destination service. (c) From System-Provided data in (CF) "Environment Variables"? If yes, then I would have expected different data, e.g. "credentials" with XSUAA properties like `clientid` and `clientsecret`. — Alexander Dümont, May 15 '20 at 09:48
I put it into notepad because the length of KeyStore didn't allow me to fit all in a screenshot. These properties are coming from an instance of the SuccessFactors Extensibility Service - automatically generated when you create the instance of the service along with a destination in the subaccount. The clientid, clientsecret etc are visible in the instance of the xsuaa service which is also bound to the application. — Inkers, May 15 '20 at 10:02
Can you log `destination.getProperty("certificates")` and share that value with us? Make sure beforehand that you do not disclose any confidential information. I cannot judge this. I am afraid without knowing which certificates the CF destination service returns you I cannot nail down the problem. — Emdee, May 15 '20 at 11:32
OK so I log it and I get the following: "logger":"com.sap.cloud.sdk.app.CandidatesServlet","thread":"http-nio-0.0.0.0-8080-exec-10","level":"INFO","categories":[],"msg":"KeyStore value:Some([ScpCfDestinationServiceV1Response.DestinationCertificate(name=sfapi_dest.p12, content=....THIS IS THE CERTIFICATE STRING, type=null} so it seems it should find it? — Inkers, May 15 '20 at 12:39
Also, when I do .get("KeyStoreLocation").toString(); the logs show "Some(sfapi_dest.p12)" and .get("KeyStorePassword").toString() logs Some(********) — Inkers, May 15 '20 at 14:20
We are investigating and have an idea. Will Update once we know more. — Emdee, May 15 '20 at 17:01

score 1 · Answer 1 · answered May 14 '20 at 10:56

1

As per the error stack trace I can derive that the SDK tries to access a key store and fails. It does so as it sees the destination properties KeyStoreLocation and KeyStorePassword. Strange is that these properties appear to be present, even though we cannot see them on the screenshot somewhere.

Guessing: Do you have accidentally or sometime in the past provided a key store and a password and for some reason the destination still has these properties set?

answered May 14 '20 at 10:56

Emdee

1,689
5
22
35

This is the confusing piece because in the cockpit if I go to myspace->service instances->sfapi_dest-> show sensitive data, I can see the keystore location, the keystore itself and the keystore pword – Inkers May 14 '20 at 11:46
Aha interesting. Yes you said you did not create the destination yourself. Can you provide a screenshot of that keystore stuff (excluding the password and sensitive data)? – Emdee May 14 '20 at 12:48
Any idea what that keystore is used for? Did you explicitly said to use that in the configuration somewhere? – Emdee May 14 '20 at 12:49
I have added the image as requested. I guess that it cannot be read from here? Perhaps I need to uncheck "Use Default JDK Trust Store" in the destination - create a certificate and upload it? – Inkers May 15 '20 at 09:32
No no, we should not mix up the notions of a key store and a trust store. – Emdee May 15 '20 at 10:10

score 1 · Answer 2 · answered May 15 '20 at 19:05

We did not expect the certificate "type" property to be empty (null) for a destination configuration, as described in your comment. The official documentation only ever mentioned "CERTIFICATE" as value.

Until the empty value can be handled in one of the next releases of the SAP Cloud SDK, please use the following workaround.

private static ScpCfHttpDestination fixKeyStore( final DestinationProperties destination )
{
    final URI uri =
        destination.get("URL", String.class).map(URI::create).getOrElseThrow(
            () -> new IllegalArgumentException("Invalid or missing \"URL\"."));

    final String name =
        destination.get("Name", String.class).getOrElseThrow(
            () -> new IllegalArgumentException("Invalid or missing \"Name\"."));

    final ScpCfHttpDestination.Builder builder = ScpCfHttpDestination.builder(name, uri);
    destination.getPropertyNames().forEach(
        propertyName -> builder.property(propertyName, destination.get(propertyName).getOrNull()));

    final Pattern pattern = Pattern.compile(".*name=(.*?)\\.(.*?), content=(.*?), type=(.*?)\\).*");
    final List<?> rawCertificates =
        destination.get("certificates", List.class).getOrElseThrow(
            () -> new IllegalArgumentException("Invalid or missing \"certificates\"."));

    rawCertificates
        .stream()
        .map(Object::toString)
        .map(pattern::matcher)
        .filter(Matcher::matches)

        // Sanity check: only apply fix in the use-case defined.
        .filter(m -> "null".equalsIgnoreCase(m.group(4)))
        .filter(m -> "p12".equalsIgnoreCase(m.group(2)))
        .map(
            m -> Try.withResources(() -> new ByteArrayInputStream(Base64.getDecoder().decode(m.group(3)))).of(
                inputStream -> {
                    final KeyStore ks = KeyStore.getInstance("PKCS12");
                    final Option<String> keyStorePassword = destination.get("KeyStorePassword", String.class);
                    ks.load(inputStream, keyStorePassword.map(String::toCharArray).getOrElse(new char[0]));
                    return ks;
                }))

        .filter(Try::isSuccess)
        .map(Try::get)
        .findFirst()
        .ifPresent(builder::keyStore);

    return builder.build();
}

Put the static fixKeyStore method into your project, e.g. inside CandidatesServlet.

Instead of calling asHttp use decorate(CandidatesServlet::fixKeyStore), e.g.

final HttpDestination destination = DestinationAccessor.getDestination(DESTINATION_NAME).decorate(CandidatesServlet::fixKeyStore);

Hi, Thanks for this. I'll try it when I get a chance and revert back. Is there anywhere where I can track this issue on github or a Jira ticket so I will know when it is resolved? — Inkers, May 15 '20 at 19:35
I will give a dedicated response to this SO question, suggesting an update of the SAP Cloud SDK to the latest version. The next scheduled release that is likely to contain a fix will be at 3rd or 4th of June, I think. — Alexander Dümont, May 18 '20 at 18:04
Please try SAP Cloud SDK Version 3.21.0 and see if the problem is solved — Emdee, Jun 04 '20 at 18:37
Hi Emdee, I have tried the latest release. I am now getting an error on the loading of the keystore - "DestinationAccessException: Failed to load key store" Further down the trace specifically the following error - "Caused by: java.io.IOException: keystore password was incorrect"..... "Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. — Inkers, Jun 08 '20 at 10:28
[This answer](https://stackoverflow.com/a/8053459/11902742) suggests that the root cause of this is most likely an invalid key. Could you please double check that? — MatKuhr, Jun 11 '20 at 09:49

DestinationAccessException on keystore location for SuccessFactors destination

2 Answers2