20

How can we make Pentaho retry rest requests on connection errors?

We have a Pentaho BI system which, among numerous data sources, is querying a particular REST api for over 20k query variations each run.

Predicatably, on most runs a few of these requests will fail due to connection hiccups. These usually manifest themselves in the Pentaho log as org.pentaho.di.core.exception.KettleException due to javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake, ultimately caused by java.io.EOFException: SSL peer shut down incorrectly.

After scouring the internet and Pentaho forums, we cannot find any instructions on adding a simple retry approach in cases such as this. Hair is being torn out.

Help me StackOverflow, you're my only hope!

Update 1: Stacktrace:

ERROR 29-11 11:02:17,659 - B - org.pentaho.di.core.exception.KettleException:
Can not result from [https://<DOMAIN>/<PATH>?<PARAMS>]
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake    
    at org.pentaho.di.trans.steps.rest.Rest.callRest(Rest.java:190)
    at org.pentaho.di.trans.steps.rest.Rest.processRow(Rest.java:385)
    at org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
    at java.lang.Thread.run(Thread.java:662)
Caused by: com.sun.jersey.api.client.ClientHandlerException: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
    at com.sun.jersey.client.apache.DefaultApacheHttpMethodExecutor.executeMethod(DefaultApacheHttpMethodExecutor.java:213)
    at com.sun.jersey.client.apache.ApacheHttpClientHandler.handle(ApacheHttpClientHandler.java:175)
    at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
    at com.sun.jersey.api.client.Client.handle(Client.java:648)
    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
    at com.sun.jersey.api.client.WebResource.get(WebResource.java:191)
    at org.pentaho.di.trans.steps.rest.Rest.callRest(Rest.java:141)
    ... 3 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:817)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:632)
    at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:827)
    at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1525)
    at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1975)
    at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:993)
    at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
    at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
    at com.sun.jersey.client.apache.DefaultApacheHttpMethodExecutor.executeMethod(DefaultApacheHttpMethodExecutor.java:210)
    ... 9 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
    at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
    ... 22 more
Daniel A. White
  • 187,200
  • 47
  • 362
  • 445
ms-tg
  • 2,688
  • 23
  • 18

2 Answers2

1

or use step error handling on the rest client step, and direct the error fields to another rest client step.

obviously that only lets you retry as many times as you have steps, but there's no point retrying more than once anyway.

Codek
  • 5,114
  • 3
  • 24
  • 38
0

Use the source, Luke.

In org.pentaho.di.trans.steps.rest.Rest.processRow(Rest.java:385) you will find 

    } catch(KettleException e) {
         boolean sendToErrorRow=false;
         String errorMessage = null;

        if (getStepMeta().isDoingErrorHandling()){
             sendToErrorRow = true;
             errorMessage = e.toString();
        } else {
            logError(BaseMessages.getString(PKG, "Rest.ErrorInStepRunning")+e.getMessage()); //$NON-NLS-1$
            setErrors(1);
            logError(Const.getStackTracker(e));
            stopAll();
            setOutputDone();  // signal end to receiver(s)
            return false;
        }

        if (sendToErrorRow) {
           // Simply add this row to the error row
           putError(getInputRowMeta(), r, 1, errorMessage, null, "Rest001");
        }

    }

It appears that you can configure error-handling on a row-basis. This seems to be what you are after.

mseebach
  • 971
  • 2
  • 9
  • 15
  • Thanks for this answer, but although this is relevant source code, it doesn't answer the question of how to do retries in Pentaho data integration logic... – ms-tg Nov 30 '11 at 15:34
  • 2
    Also, great gandalf quote from star trek ;) – ms-tg Nov 30 '11 at 16:04