13

I try to get the page content of a https url that throws an exception while getting input stream.

String httpsURL = "https://careers.virtusa.com/";
URL myurl = new URL(httpsURL);
HttpsURLConnection con = (HttpsURLConnection)myurl.openConnection();
InputStream ins = con.getInputStream();

The exception is as below,

Exception in thread "main" javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
    at com.sun.net.ssl.internal.ssl.InputRecord.handleUnknownRecord(InputRecord.java:523)
    at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:355)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)    
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1165)    
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1149)    
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:434)    
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:166)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172)    
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:234)      
    at url.JavaHttpsExample.main(JavaHttpsExample.java:18)    

Both HttpURLConnection and HttpsURLConnection fail. I tried org.apache.http.impl.client.CloseableHttpClient but getting the same exception. In browser it works fine.

itsraja
  • 1,640
  • 4
  • 32
  • 48
  • What Java version are you using? With `1.8.0_77` I get a different error, `javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target`. – dimo414 May 11 '16 at 04:37
  • That is likely your problem then; ["*Java 6 reached the end of its supported life in February 2013, at which time all public updates, including security updates, were scheduled to be stopped.*"](https://en.wikipedia.org/wiki/Java_version_history#Java_SE_6) – dimo414 May 11 '16 at 05:54
  • It is very easy to understand, I don't know why people still looking for a solution, java 6 does not support TLSV1.1 or TLSV1.2 (virtusa only support that), that is the reason for "Unrecognized SSL message" error, the protocol is not understood, JVM up to 6 supports both protocols but you need to accept (or ignore it, insecurelly) the virtusa public key to act as trusted client, if not, you will get a cert validation exception, simply. @itsraja, don't keep looking for the holy grail... – vzamanillo May 11 '16 at 09:13

7 Answers7

3

I don't get the plaintext connection error you get. I also verified that the site you are trying to comb is in fact configured for HTTPS. This is the exception which I get:

java.security.cert.CertificateException: No subject alternative DNS name matching sub.website.com found.

One solution to this problem is to install a host verifier which trusts all hosts (including sun.website.com). Try using the following code snippet which will output your target page to the Java console:

HostnameVerifier allHostsValid = new HostnameVerifier() {
    public boolean verify(String hostname, SSLSession session) {
        return true;
    }
};

HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);

String httpsURL = "https://sub.website.com/";
URL myurl = new URL(httpsURL);
HttpsURLConnection con = (HttpsURLConnection)myurl.openConnection();

BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream()));
String input;
while ((input = br.readLine()) != null) {
    System.out.println(input);
}
br.close();

Note:

I gave this answer when the OP was using a different site. The answer is valid for the conditions given though the OP may have changed since then.

Tim Biegeleisen
  • 502,043
  • 27
  • 286
  • 360
  • I'm afraid, actual url is different. I changed it for posting here. Could you please suggest an action based on the exception. – itsraja Apr 27 '16 at 19:00
  • No, I won't. Please post the actual URL. Anyway, the error message is causedd by trying to access HTTP as HTTPS. You must be doing something wrong. – Tim Biegeleisen Apr 27 '16 at 23:17
  • Could not share the domain name and IP for some reasons. I've shared the Firebug Net output here. http://imgur.com/osRs07V – itsraja Apr 28 '16 at 06:07
  • How to handle this case programmatically. – itsraja Apr 28 '16 at 06:07
  • @itsraja If you are trying to comb this jobs site, you will need a lot more work than just accessing the home page. Really, you should be trying to learn about the search API. – Tim Biegeleisen May 03 '16 at 16:40
  • No, I just want this home page only – itsraja May 03 '16 at 16:43
3

You can find all the information about the certificate of the page you want to download in this page : certificate of careers.virtusa.com.

It says that the problem is:

  • Java 6u45: Protocol or cipher suite mismatch
  • Java 7u25: Protocol or cipher suite mismatch

To solve the problem you can relax the certification or configure the client to have a protocol and a cipher that match the certificate. For example they can be:

Protocol: TLS 1.2
Cipher: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

How to solve and get the content

I write a code using apache httpcomponents, it basically trusts any source and doesn't care about the certification.

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.2</version>
</dependency>

It return the content of the page.

import java.io.*;
import org.apache.http.*;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.ssl.*;
import org.apache.http.impl.client.*;
import org.apache.http.ssl.SSLContextBuilder;

public class Main {
  public static void main(String[] args) throws Exception {
    String httpsURL = "https://careers.virtusa.com";

    SSLContextBuilder builder = new SSLContextBuilder();
    builder.loadTrustMaterial(null, new TrustSelfSignedStrategy());
    SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(
            builder.build());
    CloseableHttpClient httpclient = HttpClients.custom().setSSLSocketFactory(
            sslsf).build();

    HttpGet httpget = new HttpGet(httpsURL);
    HttpResponse response = httpclient.execute(httpget);

    HttpEntity httpEntity = response.getEntity();
    InputStream inputStream = httpEntity.getContent();
    InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
    BufferedReader bufferedReader = new BufferedReader(inputStreamReader);

    StringBuffer result = new StringBuffer();
    String line = "";
    while ((line = bufferedReader.readLine()) != null) {
      result.append(line);
    }
    System.out.println(result.toString());
  }
}

It print:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transiti ...
Troncador
  • 3,356
  • 3
  • 23
  • 40
  • I know it is working for other URLs. The problem is that this particular URL is working in browser but throws exception here. At which point switching to http is happening and how to handle it properly is the question. – itsraja May 04 '16 at 05:22
  • I test the command with careers.virtusa.com and it doesn't have certificate – Troncador May 04 '16 at 05:43
  • I have a solution for your problem – Troncador May 04 '16 at 06:05
  • Got Exception in thread "main" javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? at HttpResponse response = httpclient.execute(httpget); I tried with older httpclient I'll try with the latest. – itsraja May 04 '16 at 07:30
  • copy and paste my code, and the dependency. I run the code without any problem – Troncador May 04 '16 at 10:17
  • same exception with httpclient 4.5 also. SSLContextBuilder is depreacated. – itsraja May 04 '16 at 12:06
  • the library apache has 2 class with the same name SSLContextBuilder, you choose the old one. I change my answer and I put the complete code , with the imports – Troncador May 04 '16 at 13:34
  • jdk1.8.0_73 . I can send to you a maven project with the code – Troncador May 11 '16 at 11:52
  • You should get no point for your answer, You've changed your answer consecutively copying from the responses of other people in this question, that's not fair. – vzamanillo May 12 '16 at 10:11
  • I didn't copy, I investigate more, I found this page https://www.ssllabs.com/ and studied the certificate. And mainly my code worked from the beginning – Troncador May 12 '16 at 10:17
  • That's not true, your code does not work from the beginnig, it works by pure luck because you are running it with Java 8, but you didn't know why. – vzamanillo May 12 '16 at 10:51
  • humn... maybe you are right, then you should have put me -1 before the end of the bounty. I will test my code with java-7 – Troncador May 12 '16 at 10:56
  • I have no reasons to do a downvote for your answer, but also, there is no reason for you to get the points as best answer – vzamanillo May 12 '16 at 11:07
3

Specify SSL version that careers.virtusa.com is using. i.e. TLSv1.2 in the code.

SSLContext sc = SSLContext.getInstance("TLSv1.2");

public String getData(String URL)
{
    TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() {
        public java.security.cert.X509Certificate[] getAcceptedIssuers() {
            return null;
        }

        public void checkClientTrusted(X509Certificate[] certs,
                String authType) {
        }

        public void checkServerTrusted(X509Certificate[] certs,
                String authType) {
        }

    } };

    String output = "";
    try{

        //System.setProperty("https.proxyHost", "<PROXY HOST IP>");   // Uncomment if using proxy
        //System.setProperty("https.proxyPort", "<PROXY HOST PORT>");        // Uncomment if using proxy
        SSLContext sc = SSLContext.getInstance("TLSv1.2");
        sc.init(null, trustAllCerts, new java.security.SecureRandom());
        HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());

        // Create all-trusting host name verifier
        HostnameVerifier allHostsValid = new HostnameVerifier() {
            public boolean verify(String hostname, SSLSession session) {
                return true;
            }
        };
        // Install the all-trusting host verifier
        HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);
        /*
         * end of the fix
         */

        URL url = new URL(URL);
        URLConnection con = url.openConnection();

        InputStream ins = con.getInputStream();
        InputStreamReader isr = new InputStreamReader(ins);
        BufferedReader in = new BufferedReader(isr);

        String inputLine;


        while ((inputLine = in.readLine()) != null) {
            output = output + inputLine;
        }

        System.out.println(output);
        in.close();

    }   
    catch(Exception e){
        e.printStackTrace();            
    }


    return output;
}
Kunal Surana
  • 659
  • 5
  • 14
3

I think that the problem is the redirects (302). The last page is http not https.

Set

HttpsURLConnection.setFollowRedirects(false);

Call the first URL, get the Location header, do another call to the especified location. Get the cookie in the second call on the Set-Cookie header and again the Location header. Finally open another URLConnection with the second redirected URL informing the cookie (con.setRequestProperty("Cookie", COOKIE)).
That should work.

    HttpsURLConnection.setFollowRedirects(false);        
    String url = "https://<host>:443/OA_HTML/IrcVisitor";
    String cookie = "";
    HttpURLConnection con = (HttpURLConnection)new URL(url).openConnection();
    con.setRequestMethod("GET");
    con.connect();
    // aasuming that is always a redirect // if(con.getResponseCode() == 302) {
    url = con.getHeaderField("Location");
    con.disconnect();

    con = (HttpsURLConnection)new URL(url).openConnection();
    con.setRequestMethod("GET");
    con.connect();
    // aasuming that is always a redirect // if(con.getResponseCode() == 302) {
    url = con.getHeaderField("Location");
    cookie = con.getHeaderField("Set-Cookie");
    cookie = cookie.substring(0, cookie.indexOf(';'));

    con = (HttpsURLConnection)new URL(url).openConnection();
    con.setRequestMethod("GET");
    con.setRequestProperty("Cookie", cookie);
    con.setDoInput(true);
    con.connect();
    if(con.getResponseCode() == 200) {
        //readStream(con.getInputStream());
        //if you need to download something with a relative path from this page use "Content-Location" as base path
        String basePath = con.getHeaderField("Content-Location");
    }
fhofmann
  • 847
  • 7
  • 15
3

All requests to https://careers.virtusa.com are redirected to a McAfee Web Gateway (a proxy)

Request URL:https://careers.virtusa.com/
Request Method:GET
Status Code:302 Found
Remote Address:203.62.173.60:443

If you do a simply request to that address http://203.62.173.60:443 you will get a handshake error

Handshake failed
The SSL handshake could not be performed.
Host: 10.4.190.60
Reason: :state 21:Application response 500 handshakefailed

because the gateway are expecting a secure HTTP request from a trusted client, with the careers.virtusa.com certificate.

The problem does not appear within the Web Browser because I suppose that the front Web server of virtusa internaly redirects to the Web Gateway using the trusted certificate so finally returns the webpage without problems.

In the other hand, most of modern web browsers uses TLS 1.1 or TLS 1.2 by default to do the secure requests, but Java dont, depends on the Java version.

If you analyze careers.virtusa.com you will see that only supports 1.1 and 1.2

TLS 1.2 Yes
TLS 1.1 Yes
TLS 1.0 No
SSL 3   No
SSL 2   No

JDK 5 and 6 supports SSLv3 and TLSv1 so if you use that version you will get a SSL exception.

JDK 7, 8 and 9 supports SSLv3, TLSv1, TLSv1.1 and TLSv1.2 but you need explicitly indicate the supported protocols for your connection, in this case

new String[] { "TLSv1.1", "TLSv1.2" }

So, you need:

  • The certificate with the public key of careers.virtusa.com (using openssl s_client or from the browser directly)
  • Import a certificate into a keystore to use it as truststore with your HTTP client connection.
  • Java version > 6.
  • Set the supported protocols for the connection.

An example (with Apache HttpComponents 4.4.1)

import java.io.File;

import javax.net.ssl.SSLContext;

import org.apache.commons.io.IOUtils;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.ssl.SSLContexts;
import org.apache.http.util.EntityUtils;

public class SSLTest {

    public final static void main(String[] args) throws Exception {

        SSLContext sslcontext = SSLContexts.custom()
                .loadTrustMaterial(new File("/tmp/careers.virtusa.com.jks"), "changeit".toCharArray(), new TrustSelfSignedStrategy()).build();

        SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslcontext, new String[] { "TLSv1.1", "TLSv1.2" }, null,
                SSLConnectionSocketFactory.getDefaultHostnameVerifier());

        CloseableHttpClient httpClient = HttpClients.custom().setSSLSocketFactory(sslsf).build();
        try {

            HttpGet httpget = new HttpGet("https://careers.virtusa.com/");

            CloseableHttpResponse response = httpClient.execute(httpget);
            try {
                HttpEntity entity = response.getEntity();
                if (entity != null) {
                    System.out.println("Response content length: " + entity.getContentLength());
                    System.out.println(IOUtils.toString(entity.getContent()));
                }

                EntityUtils.consume(entity);
            } finally {
                response.close();
            }
        } finally {
            httpClient.close();
        }
    }
}

Then you can get the web page contents using both urls, http://careers.virtusa.com/ or https://careers.virtusa.com/

Hope this helps.

vzamanillo
  • 9,905
  • 1
  • 36
  • 56
2

If you trust the end-point you could disable certificate validation completely using option 2 listed here:

telling java to accept self-signed ssl certificate

Just add this before your code:

    TrustManager[] trustAllCerts = new TrustManager[] {
        new X509TrustManager() {
            public java.security.cert.X509Certificate[] getAcceptedIssuers() {
                return new java.security.cert.X509Certificate[0];
            }
            public void checkClientTrusted(
                java.security.cert.X509Certificate[] certs, String authType) {
                }
            public void checkServerTrusted(
                java.security.cert.X509Certificate[] certs, String authType) {
            }
        }
    };

    SSLContext sc = SSLContext.getInstance("SSL");
    sc.init(null, trustAllCerts, new java.security.SecureRandom());
    HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
Community
  • 1
  • 1
Zeki
  • 5,107
  • 1
  • 20
  • 27
  • I've replicated this and verified this answer enables Java 8 to open an `InputStream` connected to `https://careers.virtusa.com/`. – dimo414 May 11 '16 at 04:34
0

According to the "SSL handshake exception: unable to find valid certification path to requested target" question in security stackexchange, you need either

  1. add the missing certificate as trusted to Java
  2. or add the missing certificate to the chain of the web server.

I tried the first option from Mkyong's tutorial, and it worked.

Note: InstallCert has moved to github.

You may need to create a temp project running this java file with your target site (in your case, "careers.virtusa.com") as the arguments, see screenshot. Interact with the console, it will generate a file named "jssecacerts". Simply copy and paste this file into your "$JAVA_HOME\jre\lib\security" (in my case, "C:\Program Files\Java\jdk1.8.0_60\jre\lib\security") folder.

Finally run your app, you will get the content!

鄭脈龍
  • 336
  • 1
  • 10
  • [SSL handshake exception: unable to find valid certification path to requested target](http://security.stackexchange.com/a/78690) – 鄭脈龍 May 11 '16 at 07:30
  • I can not post more than two links in my answer, so I comment them. – 鄭脈龍 May 11 '16 at 07:34
  • My JAVA Execution environment is [J2SE-1.5] and my HTTP client is [google-http-client](https://developers.google.com/api-client-library/java/google-http-java-client/setup#google-http-client) library. – 鄭脈龍 May 11 '16 at 08:17