Java Post Query to bypass the agreement page and access other pages

Question

I am trying to automate some stuff, but recently a website has put an agreement barrier that needs to be agreed in order to access the other pages and files.

https://www.paoilandgasreporting.state.pa.us/publicreports/Modules/DataExports/DataExports.aspx

I am not able to bypass it using java post query code below, all I get is the html content of that agreeement barrier page:

HttpClient httpClient = new HttpClient();
        httpClient.getParams().setParameter(
                ClientPNames.ALLOW_CIRCULAR_REDIRECTS, true);

        PostMethod postMethod = new PostMethod(
                "https://www.paoilandgasreporting.state.pa.us/publicreports/Modules/Welcome/Welcome.aspx") {
            @Override
            public boolean getFollowRedirects() {
                return true;
            }
        };

        postMethod.addParameter("ctl00$MainContent$AgreeButton", "Agree");



        try {
            httpClient.executeMethod(postMethod);
        } catch (HttpException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

        if (postMethod.getStatusCode() == HttpStatus.SC_OK) {
            String resp = postMethod.getResponseBodyAsString();
            System.out.println(resp);
        } else {
            // ...postMethod.getStatusLine();
        }

I got the post parameters using firebug, even I tried adding the other two post parameters, even that doesn't help

//postMethod.addParameter("__EVENTVALIDATION","%2FwEWAwLA5%2FC9CwLuhr%2FJBgKws9WDC7O5VJxkt6T7bUFREi%2BvwFPh%2FDo37JKON9tdbF4jnCXF");
//  postMethod.addParameter ("__VIEWSTATE","%2FwEPDwUKMTAxODQ0NzY0Nw9kFgJmD2QWAgIDD2QWBAIBDxYCHgRUZXh0BdgQPGRpdiBzdHlsZT0iaGVpZ2h0OiAxNDlweDsgd2lkdGg6IDk1MnB4OyBiYWNrZ3JvdW5kLXJlcGVhdDogbm8tcmVwZWF0OyBiYWNrZ3JvdW5kLWltYWdlOnVybChodHRwczovL3d3dy5haHMyLmRlcC5zdGF0ZS5wYS51cy9pbWFnZXMvZGVwX2V4dGVybmFsX21haW5fYmFubmVyXzIuanBnKSI%2BCgoJPGEgc3R5bGU9ImhlaWdodDoxMDBweDsgd2lkdGg6MzkwcHg7IGxlZnQ6MHB4OyB0b3A6MHB4OyBwb3NpdGlvbjpyZWxhdGl2ZTsgZGlzcGxheTpibG9jazsgdGV4dC1kZWNvcmF0aW9uOm5vbmU7IiB0aXRsZT0iREVQIEhvbWUiIGhyZWY9Imh0dHA6Ly93d3cuZGVwd2ViLnN0YXRlLnBhLnVzLyI%2BCgkJPGVtIHN0eWxlPSJ2aXNpYmlsaXR5OiBoaWRkZW47IGNvbG9yOiAjMDAwMDAwOyBiYWNrZ3JvdW5kLWNvbG9yOiAjRkZGRkZGOyI%2BQWdlbmN5PC9lbT4KCTwvYT4KCTxhIHN0eWxlPSJoZWlnaHQ6MzBweDsgd2lkdGg6MTYwcHg7IGxlZnQ6OHB4OyB0b3A6MTJweDsgcG9zaXRpb246cmVsYXRpdmU7IGRpc3BsYXk6YmxvY2s7IHRleHQtZGVjb3JhdGlvbjpub25lOyIgdGl0bGU9IlBlbm5zeWx2YW5pYSIgaHJlZj0iaHR0cDovL3BhLmdvdi8iPgoJCTxlbSBzdHlsZT0idmlzaWJpbGl0eTogaGlkZGVuOyBjb2xvcjogIzAwMDAwMDsgYmFja2dyb3VuZC1jb2xvcjogI0ZGRkZGRjsiPlBlbm5zeWx2YW5pYTwvZW0%2BCgk8L2E%2BCgoJPGEgc3R5bGU9ImhlaWdodDoxNnB4OyB3aWR0aDoxNDBweDsgbGVmdDo2MjBweDsgdG9wOi0xOHB4OyBwb3NpdGlvbjpyZWxhdGl2ZTsgZGlzcGxheTpibG9jazsgdGV4dC1kZWNvcmF0aW9uOm5vbmU7IHRleHQtYWxpZ246Y2VudGVyOyBjb2xvcjogI0ZGRkZGRjsgYmFja2dyb3VuZC1jb2xvcjogIzE2M0Y2RDsgZm9udC1zaXplOiAwLjhlbTsgZm9udC13ZWlnaHQ6IGJvbGQ7IGZvbnQtZmFtaWx5OiBWZXJkYW5hLCBBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmOyIgaHJlZj0iaHR0cDovL3d3dy5wb3J0YWwuc3RhdGUucGEudXMvcG9ydGFsL3NlcnZlci5wdD9vcGVuPTUxMiZvYmpJRD0yMDU5OCZQYWdlSUQ9MTA1MjE2MCZtb2RlPTIiPgoJCVBBIFNUQVRFIEFHRU5DSUVTCgk8L2E%2BCgoJPGEgc3R5bGU9ImhlaWdodDoxNnB4OyB3aWR0aDoxNjBweDsgbGVmdDo3ODBweDsgdG9wOi0zNHB4OyBwb3NpdGlvbjpyZWxhdGl2ZTsgZGlzcGxheTpibG9jazsgdGV4dC1kZWNvcmF0aW9uOm5vbmU7IHRleHQtYWxpZ246Y2VudGVyOyBjb2xvcjogI0ZGRkZGRjsgYmFja2dyb3VuZC1jb2xvcjogIzE2M0Y2RDsgZm9udC1zaXplOiAwLjhlbTsgZm9udC13ZWlnaHQ6IGJvbGQ7IGZvbnQtZmFtaWx5OiBWZXJkYW5hLCBBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmOyIgaHJlZj0iaHR0cDovL3d3dy5wb3J0YWwuc3RhdGUucGEudXMvcG9ydGFsL3NlcnZlci5wdD9vcGVuPTUxMiZvYmpJRD0yMDU5OCZQYWdlSUQ9MTA1MjE1OCZtb2RlPTIiPgoJCU9OTElORSBTRVJWSUNFUwoJPC9hPgoKCTxhIHN0eWxlPSJoZWlnaHQ6MTZweDsgd2lkdGg6MTYwcHg7IGxlZnQ6NjE1cHg7IHRvcDotMjlweDsgcG9zaXRpb246cmVsYXRpdmU7IGRpc3BsYXk6YmxvY2s7IHRleHQtZGVjb3JhdGlvbjpub25lOwl0ZXh0LWFsaWduOmNlbnRlcjsgY29sb3I6ICNGRkZGRkY7IGZvbnQtc2l6ZTogMC44ZW07IGZvbnQtZmFtaWx5OiBWZXJkYW5hLCBBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmOyIgaHJlZj0iaHR0cDovL3d3dy5nb3Zlcm5vci5wYS5nb3YiIHRpdGxlPSJUb20gQ29yYmV0dCwgR292ZXJub3IiPlRvbSBDb3JiZXR0LCBHb3Zlcm5vcgoJPC9hPgoJPGEgc3R5bGU9ImhlaWdodDoxNnB4OyB3aWR0aDoxOTBweDsgbGVmdDo3NjVweDsgdG9wOi00NXB4OyBwb3NpdGlvbjpyZWxhdGl2ZTsgZGlzcGxheTpibG9jazsgdGV4dC1kZWNvcmF0aW9uOm5vbmU7CXRleHQtYWxpZ246Y2VudGVyOyBjb2xvcjogI0ZGRkZGRjsgZm9udC1zaXplOiAwLjhlbTsgZm9udC1mYW1pbHk6IFZlcmRhbmEsIEFyaWFsLCBIZWx2ZXRpY2EsIHNhbnMtc2VyaWY7IiBocmVmPSJodHRwOi8vd3d3LmRlcHdlYi5zdGF0ZS5wYS51cy9zZWNyZXRhcnkiIHRpdGxlPSJNaWNoYWVsIEtyYW5jZXIsIFNlY3JldGFyeSI%2BTWljaGFlbCBLcmFuY2VyLCBTZWNyZXRhcnkKCTwvYT4KCQkKPC9kaXY%2BZAIFDxYCHwAFxgg8ZGl2IHN0eWxlPSJ3aWR0aDogMTAwJTsgdGV4dC1hbGlnbjogY2VudGVyOyBtYXJnaW46IDAgYXV0bztmb250LXNpemU6IDAuOWVtOyBsaW5lLWhlaWdodDogMTZweDsgY29sb3I6ICMwMzM2Njg7IGZvbnQtZmFtaWx5OiBWZXJkYW5hLCBBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmOyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiBub3JtYWw7IGZvbnQtdmFyaWFudDogbm9ybWFsOyB0ZXh0LWRlY29yYXRpb246IG5vbmU7Ij4KCTxhIHN0eWxlPSJmb250LXNpemU6IDAuOWVtOyBsaW5lLWhlaWdodDogMTZweDsgY29sb3I6ICMwMzM2Njg7IGZvbnQtZmFtaWx5OiBWZXJkYW5hLCBBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmOyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiBub3JtYWw7IGZvbnQtdmFyaWFudDogbm9ybWFsOyB0ZXh0LWRlY29yYXRpb246IG5vbmU7Imh0dHA6Ly93d3cucG9ydGFsLnN0YXRlLnBhLnVzL3BvcnRhbC9zZXJ2ZXIucHQ%2Fb3Blbj01MTImb2JqSUQ9MjA1OTgmUGFnZUlEPTEwNTE2MDYmbW9kZT0yIgoJdGl0bGU9IlByaXZhY3kgUG9saWN5Ij5Qcml2YWN5IFBvbGljeTwvYT4KCSZuYnNwO3wmbmJzcDsKCTxhIHN0eWxlPSJmb250LXNpemU6IDAuOWVtOyBsaW5lLWhlaWdodDogMTZweDsgY29sb3I6ICMwMzM2Njg7IGZvbnQtZmFtaWx5OiBWZXJkYW5hLCBBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmOyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiBub3JtYWw7IGZvbnQtdmFyaWFudDogbm9ybWFsOyB0ZXh0LWRlY29yYXRpb246IG5vbmU7ImhyZWY9Imh0dHA6Ly93d3cucG9ydGFsLnN0YXRlLnBhLnVzL3BvcnRhbC9zZXJ2ZXIucHQ%2Fb3Blbj01MTImb2JqSUQ9MjA1OTgmUGFnZUlEPTEwNTE5NDAmbW9kZT0yIgoJdGl0bGU9IlNlY3VyaXR5IFBvbGljeSI%2BU2VjdXJpdHkgUG9saWN5PC9hPgkJCgk8YnIgLz4KCTxzcGFuIHN0eWxlPSJmb250LXNpemU6IDAuOWVtIj4KCQlDb3B5cmlnaHQmbmJzcDsmY29weTsyMDA5LTIwMTImbmJzcDtDb21tb253ZWFsdGggb2YgUGVubnN5bHZhbmlhLiBBbGwgUmlnaHRzIFJlc2VydmVkCgk8L3NwYW4%2BCjwvZGl2PmRkxyei%2F5tm7ky3D270eOJqk5kA63kYz4W0lQuUC%2BjoMj4%3D");

Any Idea on how to by pass these kind of pages?

Thanks for your help.

Answer 1

If you look at the HTML of the agreement page, you'll see that the form tag that the Agree button is within submits to the relative URL "Agreement.aspx". Essentially, this sets some kind of cookie, or a session variable on the server, which tells the server not to stick that page in your face again.

You will have to use the same HTTPClient instance to perform these two steps:

• agree, by submitting to Agreement.aspx
• then proceed with the POST you were already doing previously -

In order to get the session tracking in HTTPClient, you'll need to set it up a little differently:

HttpClient httpClient = new DefaultHttpClient();
CookieStore cookieStore = new BasicCookieStore();
HttpContext httpContext = new BasicHttpContext();
httpContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);

And then pass this httpContext instance into each request:

HttpResponse resp = httpClient.executeMethod(postMethod, httpContext);