Google dataflow : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Question

I have a dataflow that does a request to an API to retrieve some data. Recently there was an update on the ciphers in the API and the dataflow suddenly started failing. I was using java 1.8 and beam SDK 2.19.0. The same code is working when run locally. I tried upgrading to java 11 and beam SDK 2.24.0 just in case the new ciphers weren't supported in the version I was using but I'm getting the same result, it runs locally but I get the same error in dataflow. This is the code I'm using to do the request to the API:

        URL url = new URL(urlString);
        HttpsURLConnection con = null;
        outer: for (int retry = 0; retry <= maxRetries && !connected; retry++) {
            if (retry > 0) {
                logger.info("retry " + retry + "/" + maxRetries);
                Thread.sleep(retryDelayMs);
            }
            logger.info("Creating connection to Customer Master Read API");
            con = (HttpsURLConnection) url.openConnection();
            con.setRequestMethod("GET");
            con.setRequestProperty("Authorization", token);
            con.setRequestProperty("x-client-id", xClientId);
            con.setRequestProperty("Content-Type", "application/json");
            con.setDoOutput(true);

            responseCode = con.getResponseCode();

It fails in this line:

            responseCode = con.getResponseCode();

And this is the full error trace:

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at com.....gcp.dataflow.dpfw.http.ApiREST.getCustomerRequest(ApiREST.java:521)
    at com.....gcp.dataflow.dpfw.transform.GetCustomer$GetCustomerInfo.processElement(GetCustomer.java:105)
    at com.....gcp.dataflow.dpfw.transform.AutoValue_GetCustomer_GetCustomerInfo$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:227)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:186)
    at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:334)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
    at org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:279)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:267)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:79)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:413)
    at org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:73)
    at com.....gcp.dataflow.dpfw.util.FilteringMessage$MessageFilter.processElement(FilteringMessage.java:85)
    at com.....gcp.dataflow.dpfw.util.AutoValue_FilteringMessage_MessageFilter$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:227)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:186)
    at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:334)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
    at org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:279)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:267)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:79)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:413)
    at org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:73)
    at com.....gcp.dataflow.dpfw.transform.PubsubTransform$PubSubMessageTransformation.processElement(PubsubTransform.java:101)
    at com.....gcp.dataflow.dpfw.transform.AutoValue_PubsubTransform_PubSubMessageTransformation$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:227)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:186)
    at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:334)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
    at org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:279)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:267)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:79)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:413)
    at org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:73)
    at org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:139)
    at org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:227)
    at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:186)
    at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:334)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
    at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1365)
    at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:154)
    at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1085)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:128)
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:308)
    at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:279)
    at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:181)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
    at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1581)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
    at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:329)
    at com.....gcp.dataflow.dpfw.http.ApiREST.getCustomerRequest(ApiREST.java:464)
    ... 52 more

Any idea why this is failing? And how to solve it?

Thank you in advance!!

Possible duplicate? Checkout this question/answer here: https://stackoverflow.com/questions/6353849/received-fatal-alert-handshake-failure-through-sslhandshakeexception — hooknc, Nov 04 '20 at 17:55
my question is more related on why it's failing in dataflow and if there is a way to enable a cipher from the code or something like that. I can't add more ciphers to the environment as it's managed by google. — NaiaV, Nov 05 '20 at 08:09
And the answer is, "It's complicated". It is not understood where the problem is actually occurring. It could be the clients cipher, but generally speaking, Java doesn't just remove a cipher for no reason. It is generally because that cipher has been deprecated. Perhaps the problem is on the server side where the server is returning out of date cipher options or is using a certificate that isn't considered strong enough? SSL issues are a two way street and problems can be caused by both sides seemingly out of the blue. — hooknc, Nov 05 '20 at 18:02
Check the certificate of the server that is being called with the SSL Labs, SSL Server Test: https://www.ssllabs.com/ssltest/ — hooknc, Nov 05 '20 at 18:03
I checked and the overall rating is A+. Also if I execute the code locally with both java 8 and java 11, it works fine, so I guess the issue isn't that java doesn't support the ciphers. — NaiaV, Nov 06 '20 at 11:17

score 2 · Accepted Answer · answered Jul 26 '21 at 07:56

I spent almost 10 hours fixing the problem. I would like to thanks my team lead. When I asked for the solution. He told me we already had a fix for one of another data flow.

use : enable_conscrypt_security_provider as below

  @Override
  protected void overridePipelineOptions(final PipelineOptions options) {
    options.setJobName(JOB_NAME);
    ((DataflowPipelineOptions) options).setExperiments(Arrays.asList("enable_conscrypt_security_provider"));
  }

Google dataflow : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

1 Answers1