groovy/grails - Can't make HTTPBuilder work with a URL (provided by Paypal)

Question

PLEASE NOTE: With the code/info below, this problem can be reproduced in 2-3 minutes.

BACKGROUND: Firstly I haven't found a plugin to help with the Paypal Advanced Interface (their mid-level solution between "Standard" and "Pro"), but if if I don't need to roll my own HTTPBuilder and other interface code I'm definitely for that.

Now, I'm able to narrow down the HTTPBuilder problem to making the curl command work from the DOS prompt using the correct URL; I just can't get it to work with groovy's HTTPBuilder.

So, what works that one can easily try is:

 c:\groovy\ex>curl https://pilot-payflowpro.paypal.com
 curl: (52) Empty reply from server

But still a reply. Or the real URL to get a SECURETOKEN back from Paypal is more like:

c:\groovy\ex>curl https://pilot-payflowpro.paypal.com -kd "PARTNER=PayPal&VENDOR=ROAdvanced&USER=ROAdvanced&PWD=joespizza1&TRXTYPE=S&MODE=TEST&AMT=40&CREATESECURETOKEN=Y&SECURETOKENID=12528208de1413abc3d60c86233"
RESULT=0&SECURETOKEN=15XTWEZtFlkeNqtWCBZHSTgcs&SECURETOKENID=12528208de1413abc3d60c86233&RESPMSG=Approved

OR you'll get a result like the following, but either result is good, since Paypal is sending a response in both cases!

RESULT=7&SECURETOKENID=12528208de1413abc3d60c86233&RESPMSG=Field format error: Secure Token Id already been used

Ok, my code is as follows. Please note my code errors out with an immediate failure even though (I think) I'm using a delay:

 Class: org.apache.http.NoHttpResponseException
 Message: The target server failed to respond

This happens around the

  http.request(GET, ContentType.ANY) {

Please note even with setting the delays on HTTPBuilder, this failure happens IMMEDIATELY. I'll put the whole code/stack trace in at the end. Also note if one takes out adding the SSL security as described in this post, the result is the same error, i.e. the server failed to respond.

So the code is:

 package apps

 import grails.converters.*
 import org.codehaus.groovy.grails.web.json.*; // package containing JSONObject,  JSONArray,...
 import groovyx.net.http.*
 import static groovyx.net.http.ContentType.*
 import static groovyx.net.http.Method.*

 import javax.net.ssl.X509TrustManager
 import javax.net.ssl.SSLContext
 import java.security.cert.X509Certificate
 import javax.net.ssl.TrustManager
 import java.security.SecureRandom
 import org.apache.http.conn.ssl.SSLSocketFactory
 import org.apache.http.conn.scheme.Scheme
 import org.apache.http.conn.scheme.SchemeRegistry
 import org.apache.http.conn.ssl.X509HostnameVerifier


class PaypalController {

   def index() {
    def paypalUrl = "https://pilot-payflowpro.paypal.com?PARTNER=PayPal&VENDOR=ROAdvanced&USER=ROAdvanced&PWD=joespizza1&TRXTYPE=S&MODE=TEST&AMT=40&CREATESECURETOKEN=Y&SECURETOKENID=12528208de1413abc3d60c86cdr87"
    //def paypalUrl = "https://pilot-payflowpro.paypal.com"
    println "Making new http Builder with paypalURL ..."
    def http = new HTTPBuilder(paypalUrl)

    println "Now setting timeouts ..."
    http.getClient().getParams().setParameter("http.connection.timeout", new Integer(12000))
    http.getClient().getParams().setParameter("http.socket.timeout", new Integer(30000))

    //=== SSL UNSECURE CERTIFICATE ===

    println "Don't know if needed, but since timeouts don't work to get a response, setting up SSL bypass"

    def sslContext = SSLContext.getInstance("SSL")
    sslContext.init(null, [new X509TrustManager() {
        public X509Certificate[] getAcceptedIssuers() {null }

        public void checkClientTrusted(X509Certificate[] certs, String authType) { }

        public void checkServerTrusted(X509Certificate[] certs, String authType) { }
    }] as TrustManager[], new SecureRandom())

    //SSLSocketFactory sf = new org.apache.http.conn.ssl.SSLSocketFactory(sslContext, org.apache.http.conn.ssl.SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER)

    SSLSocketFactory sf = new org.apache.http.conn.ssl.SSLSocketFactory(sslContext)
    sf.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER)
    def httpsScheme = new Scheme("https", sf, 443)
    http.client.connectionManager.schemeRegistry.register(httpsScheme)


    println "Now doing the get"
    // perform a GET request, expecting JSON response data
    try {
        http.request(GET, ContentType.ANY) {
            println "Issued the get waiting for the response"
            // Sleeping doesn't help
            //Thread.sleep(2000)
            // println "Done sleeping, looking to process success"

            response.success = { resp, any ->
                println "in success code"
                println "My response handler got response: ${resp.statusLine}"
                println "Response length: ${resp.headers.'Content-Length'}"
                assert resp.status == 200

                def result = any.results;

                render(view: "index", model: [message: "Request sent", result: result]);
            }
            println "past the success code"
        }//end of request
    } catch (groovyx.net.http.HttpResponseException ex) {
        println "Had response exception ...."
        ex.printStackTrace()
        return null
    } catch (java.net.ConnectException ex) {
        println "Had connection exception ...."
        ex.printStackTrace()
        return null
    }

    finally {
        http.shutdown()
    }
}//end of method


def fail() {}

def success() {}

}

------------------- Running this code I get (with no delays anywhere) -------------------------

  Making new http Builder with paypalURL ...
  Now setting timeouts ...
  Don't know if needed, but since timeouts don't work to get a response, setting up SSL bypass
  Now doing the get
  Issued the get waiting for the response
  past the success code
  | Error 2013-07-19 13:27:32,301 [http-bio-8080-exec-10] ERROR errors.GrailsExceptionResolver  - NoHttpResponseException occurred when processing request: [GET] /apps/paypal/index
  The target server failed to respond. Stacktrace follows:
  Message: The target server failed to respond
  Line | Method
  ->> 101 | parseHead             in org.apache.http.impl.conn.DefaultResponseParser
  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
  |   210 | parse                 in org.apache.http.impl.io.AbstractMessageParser
  |   271 | receiveResponseHeader in org.apache.http.impl.AbstractHttpClientConnection
  |   233 | receiveResponseHeader in org.apache.http.impl.conn.DefaultClientConnection
  |   209 | receiveResponseHeader in org.apache.http.impl.conn.AbstractClientConnAdapter
  |   292 | doReceiveResponse     in org.apache.http.protocol.HttpRequestExecutor
  |   126 | execute . . . . . . . in     ''
  |   483 | execute               in org.apache.http.impl.client.DefaultRequestDirector
  |   641 | execute . . . . . . . in org.apache.http.impl.client.AbstractHttpClient
  |   576 | execute               in     ''
  |   477 | doRequest . . . . . . in groovyx.net.http.HTTPBuilder
  |   441 | doRequest             in     ''
  |   390 | request . . . . . . . in     ''
  |    56 | index                 in apps.PaypalController$$EOC4TiIi
  |   195 | doFilter . . . . . .  in grails.plugin.cache.web.filter.PageFragmentCachingFilter
  |    63 | doFilter              in grails.plugin.cache.web.filter.AbstractFilter
  |   886 | runTask . . . . . . . in java.util.concurrent.ThreadPoolExecutor$Worker
  |   908 | run                   in     ''
  ^   619 | run . . . . . . . . . in java.lang.Thread

----------------- I had to add the following to BuildConfig.groovy to get this to this work ----

...

dependencies {
    // specify dependencies here under either 'build', 'compile', 'runtime', 'test' or 'provided' scopes e.g.
    compile('org.codehaus.groovy.modules.http-builder:http-builder:0.5.2') {
            excludes "commons-logging", "xml-apis", "groovy"
    }

---------------------------- Code update per Ralf's response -------------

Code hopefully matching Groovy HTTPBuilder documentation:

NOTE THIS PRODUCES THE SAME ERROR AT THE SAME LINE, I.E.:

org.apache.http.NoHttpResponseException, Message: The target server failed to respond

at the line: http.request(POST) {

class PaypalController {

def index() {
    def paypalUrl = "https://pilot-payflowpro.paypal.com?PARTNER=PayPal&VENDOR=ROAdvanced&USER=ROAdvanced&PWD=joespizza1&TRXTYPE=S&MODE=TEST&AMT=40&CREATESECURETOKEN=Y&SECURETOKENID=12528208de1413abc3d60c86cdr87"

    println "Making new http Builder with paypalURL ..."
    def http = new HTTPBuilder(paypalUrl)

    println "Now doing the post as a request per example"
    // perform a GET request, expecting JSON response data
    try {
        http.request(POST) {
            body =  [ status : 'update!' , source : 'httpbuilder' ]
            requestContentType = ContentType.URLENC
            println "Issued the post waiting for the response"

            response.success = { resp ->
                println "in success code"
                println "My response handler got response: ${resp.statusLine}"
                println "Response length: ${resp.headers.'Content-Length'}"
                assert resp.status == 200

                //def result = any.results;

                render(view: "index", model: [message: "Request sent", result: result]);
            }
            println "past the success code"
        }//end of request
    } catch (groovyx.net.http.HttpResponseException ex) {
        println "Had response exception ...."
        ex.printStackTrace()
        return null
    } catch (java.net.ConnectException ex) {
        println "Had connection exception ...."
        ex.printStackTrace()
        return null
    }

    finally {
        http.shutdown()
    }
}//end of method

def fail() {}
def success() {}
}

It's just a get reqest. Why don't you use 'new URL(yourURL).text'? And another thought... Could it be that you are behind a proxy? — rdmueller, Jul 21 '13 at 07:50
I'm on a computer at my house so I don't think so. Besides curl hits the url fine. Your suggestion gave a different error on the .text call. I got a java.net.SocketException call, "Unexpected end of file from server". I copied this url from a curl test so it should be fine, so ? Also the HTTPBuilder docs show in passing in a text url as I am, so ? This code is very fast to try - perhaps you can see how it works on your machine and if any other insights pop up. Thanks. — Ray, Jul 22 '13 at 00:24

score 2 · Accepted Answer · answered Jul 22 '13 at 05:08

2

found your problem: curl -d is not a GET but a POST request. If you leave out the -d in your curl request, you'll run in to the same problem as with your groovy...

Try this groovy code:

def url = new URL("https://pilot-payflowpro.paypal.com")
def body= "PARTNER=PayPal&VENDOR=ROAdvanced&USER=ROAdvanced&PWD=..."
def connection = url.openConnection()
connection.setRequestMethod("POST")
connection.doOutput = true

def writer = new OutputStreamWriter(connection.outputStream)
writer.write(body)
writer.flush()
writer.close()
connection.connect()
def response = connection.content.text

answered Jul 22 '13 at 05:08

rdmueller

10,742
10
69
126

PS: I guess your code will work too, if you switch from "GET" to "POST" – rdmueller Jul 22 '13 at 07:31
Surprisingly shifting to a http.request(POST, ContentType.ANY) { produced the exact same result. – Ray Jul 22 '13 at 18:28
sorry, by "switch GET to POST" I meant you should rework your code in order to change from a GET-Request to a POST one. This involves changing the way you submit your data. see http://groovy.codehaus.org/modules/http-builder/doc/post.html for more details. btw: you can trow away half of your code (the part with the timeouts and the insecure certificate). And give my code a try - it works and seems to be simpler :-) – rdmueller Jul 22 '13 at 19:38
Thanks Ralf your groovy code did produce a response. I will look at the reference you gave and re-write. – Ray Jul 22 '13 at 19:43
A bit confused: examples 2 and 3 in the post you gave look very much similar to the GET request code. Doing this still doesn't work with the POST for some reason, I'll update the code above. – Ray Jul 22 '13 at 19:51
With the http builder, posts easily look like get requests. In this case the line ,body=' makes the difference... – rdmueller Jul 22 '13 at 20:40
Just added the example (2) body and requestContentType line and get a null pointer exception on the "body =" line, please see updated code above. Any idea why I get this null pointer exception? – Ray Jul 22 '13 at 21:40
hm. your updated code is still wrong. The parameters from the get request (everything behind the '?' in the paypalUrl) have to be transformed into the 'body=' map. BTW: I tried to get the http-builder to work with your url, but it seemed that the response is a problem for the builder. At least I succeeded in getting a 200 as response code, but with no result (it was null) :-( – rdmueller Jul 23 '13 at 17:30
Ralf I am processing your comment. I noticed the second post example shown in the HTTPBuilder documentation (you pointed to above) also gives a null pointer exception. – Ray Jul 23 '13 at 18:23
I still wonder why you don't want to go for my simplified code :-) – rdmueller Jul 23 '13 at 19:00
Oh I see what you mean (i.e. skip HTTPBuilder). I am working on that now. Any recommendations on parsing the response, any fast/easy ways to grab this stuff as HTTPBuilder seems to provide. – Ray Jul 23 '13 at 20:16
the response can be parsed with `JSON.parse(response)`. the needed imports are already in your code. – rdmueller Jul 23 '13 at 20:44
Fantastic thanks Ralf -- Hey, check out Graeme Rocher's (the Lead for Grails) response to my question -- he just responded. I'm trying to only swear under my breath!!!. Thanks much for your help dude! Graeme's response:http://grails.1312388.n4.nabble.com/HTTPBuilder-fails-following-the-exact-example-in-its-documentation-td4647120.html#a4647123 – Ray Jul 23 '13 at 20:52

score 1 · Answer 2 · answered Jul 23 '13 at 17:32

Here is the http-builder code with a proper body:

    @Grab(group='org.codehaus.groovy.modules.http-builder',module='http-builder', version='0.5.0')
     import grails.converters.*
     import org.codehaus.groovy.grails.web.json.*; // package containing JSONObject,  JSONArray,...
     import groovyx.net.http.*
     import static groovyx.net.http.ContentType.*
     import static groovyx.net.http.Method.*

    class PaypalController {

    def index() {
        def paypalUrl = "https://pilot-payflowpro.paypal.com"

        println "Making new http Builder with paypalURL ..."
        def http = new HTTPBuilder(paypalUrl)

        println "Now doing the post as a request per example"
        // perform a GET request, expecting JSON response data
        try {
            http.request(POST,JSON) {
                body =  [ 
                    PARTNER:'PayPal',
                    VENDOR:'yyy',
                    USER:'yyy',
                    PWD:'xxx',
                    TRXTYPE:'S',
                    MODE:'TEST',
                    AMT:'40',
                    CREATESECURETOKEN:'Y',
                    SECURETOKENID:'xxx'
                ]
                requestContentType = ContentType.URLENC
                println "Issued the post waiting for the response"

                response.success = { resp ->
                    println "in success code"
                    println "My response handler got response: ${resp.statusLine}"
                    println "Response length: ${resp.headers.'Content-Length'}"
                    assert resp.status == 200

                    //def result = any.results;

                    render(view: "index", model: [message: "Request sent", result: result]);
                }
                println "past the success code"
            }//end of request
        } catch (groovyx.net.http.HttpResponseException ex) {
            println "Had response exception ...."
            ex.printStackTrace()
            return null
        } catch (java.net.ConnectException ex) {
            println "Had connection exception ...."
            ex.printStackTrace()
            return null
        }

        finally {
            http.shutdown()
        }
    }//end of method
    }
    def controller = new PaypalController()
    controller.index()

the @Grab statement lets you execute this in the groovyConsole...

Thanks Ralf however my GroovyConsole which should be the latest one or one of the latest (version 2.1.3) won't do the @Grab without generating an exception. Thusly I copied your code into my Grails controller and still get same null exception on the body = statement. I also get this same null exception when I try the 2nd example from the Grails HTTPBuilder documentation reference that you supplied (i.e. null exception at body = line). I wonder what's wrong. I'm bringing in HTTPBuilder as described at the bottom of my first main question submission (i.e before we started talking). Thanks — Ray, Jul 23 '13 at 18:48
I had this null pointer exceptions at the body when I used `http.request(POST) {` instead of `http.request(POST,JSON) {` — rdmueller, Jul 23 '13 at 18:59
Can you also please try executing the second example code in the link you gave, i.e. http://groovy.codehaus.org/modules/http-builder/doc/post.html. This too gives me a null pointer exception. — Ray, Jul 23 '13 at 19:08
Fantastic thanks Ralf -- Hey, check out Graeme Rocher's (the Lead for Grails) response to my question -- he just responded. I'm trying to only swear under my breath!!!. Thanks much for your help dude! Graeme's response:http://grails.1312388.n4.nabble.com/HTTPBuilder-fails-following-the-exact-example-in-its-documentation-td4647120.html#a4647123 — Ray, Jul 23 '13 at 20:54

groovy/grails - Can't make HTTPBuilder work with a URL (provided by Paypal)

---------------------------- Code update per Ralf's response -------------

2 Answers2