Java parse url error

Question

I am trying to connect to a url and keep getting a 403 error. Below is the 2 sets of code I used to connect but it fails. do help me out! thanks!

code block 1

URL urll = new URL("https://cws.hellogold.com/api/v2/spot_price.json");             
URLConnection yc = urll.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));

code block 2

HttpURLConnection httpURLConnection = (HttpURLConnection) urll.openConnection();
        InputStream inputStream = httpURLConnection.getInputStream();
        BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));

error

java.io.IOException: Server returned HTTP response code: 403 for URL: https://cws.hellogold.com/api/v2/spot_price.json
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)

[Might not have anything to do with your code](https://en.wikipedia.org/wiki/HTTP_403)... — Cardinal System, Dec 24 '17 at 04:07
I can access the end point https://cws.hellogold.com/api/v2/spot_price.json. Perhaps the website has usage limit. — ThomasEdwin, Dec 24 '17 at 04:11
your URL uses HTTPS. You can check one solution here: https://stackoverflow.com/questions/42805652/setup-asynchttpclient-to-use-https — user1506104, Dec 24 '17 at 04:15

score 2 · Answer 1 · answered Dec 24 '17 at 04:22

This happens because the owner of the file has set up the site to disallow queries that do not have an acceptable header. Their message is The owner of this website (cws.hellogold.com) has banned your access based on your browser's signature (3d20bd6848756e32-ua21).

Here is the program that showed me that this is the case:

import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Scanner;

public class JavaHello {
    public static void main(String[] args) throws Exception {
        String response;
        URL url = new URL("https://cws.hellogold.com/api/v2/spot_price.json");
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();
        connection.getResponseCode();
        InputStream stream = connection.getErrorStream();
        if (stream == null) 
            stream = connection.getInputStream();
        try (Scanner scanner = new Scanner(stream)) {
            scanner.useDelimiter("\\Z");
            response = scanner.next();
        }
        System.out.println(response);
    }
}

This is the error page that was returned:

/usr/lib/jvm/java-8-oracle/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:39597,suspend=y,server=n -javaagent:/opt/idea-IU-173.3727.127/lib/rt/debugger-agent.jar=/tmp/capture.props -Dfile.encoding=UTF-8 -classpath /usr/lib/jvm/java-8-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-8-oracle/jre/lib/deploy.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-oracle/jre/lib/javaws.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfxswt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-8-oracle/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-oracle/jre/lib/plugin.jar:/usr/lib/jvm/java-8-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar:/mnt/_/work/experiments/scala/stackOverflow/target/scala-2.12/classes:/home/mslinn/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4.jar:/opt/idea-IU-173.3727.127/lib/idea_rt.jar JavaHello
Connected to the target VM, address: '127.0.0.1:39597', transport: 'socket'
Disconnected from the target VM, address: '127.0.0.1:39597', transport: 'socket'
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Access denied | cws.hellogold.com used Cloudflare to restrict access</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>
<!--[if lte IE 9]><script type="text/javascript" src="/cdn-cgi/scripts/jquery.min.js"></script><![endif]-->
<!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/zepto.min.js"></script><!--<![endif]-->
<script type="text/javascript" src="/cdn-cgi/scripts/cf.common.js"></script>

</head>
<body>
  <div id="cf-wrapper">
    <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
    <div id="cf-error-details" class="cf-error-details-wrapper">
      <div class="cf-wrapper cf-header cf-error-overview">
        <h1>
          <span class="cf-error-type" data-translate="error">Error</span>
          <span class="cf-error-code">1010</span>
          <small class="heading-ray-id">Ray ID: 3d20bd6848756e32 &bull; 2017-12-24 04:15:30 UTC</small>
        </h1>
        <h2 class="cf-subheadline" data-translate="error_desc">Access denied</h2>
      </div><!-- /.header -->

      <section></section><!-- spacer -->

      <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
          <div class="cf-column">
            <h2 data-translate="what_happened">What happened?</h2>
            <p>The owner of this website (cws.hellogold.com) has banned your access based on your browser's signature (3d20bd6848756e32-ua21).</p>
          </div>


        </div>
      </div><!-- /.section -->

      <div class="cf-error-footer cf-wrapper">
  <p>
    <span class="cf-footer-item">Cloudflare Ray ID: <strong>3d20bd6848756e32</strong></span>
    <span class="cf-footer-separator">&bull;</span>
    <span class="cf-footer-item"><span data-translate="your_ip">Your IP</span>: 73.241.128.88</span>
    <span class="cf-footer-separator">&bull;</span>
    <span class="cf-footer-item"><span data-translate="performance_security_by">Performance &amp; security by</span> <a data-orig-proto="https" data-orig-ref="www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
</div><!-- /.error-footer -->


    </div><!-- /#cf-error-details -->
  </div><!-- /#cf-wrapper -->

  <script type="text/javascript">
  window._cf_translation = {};


</script>

</body>
</html>

The solution is to specify a User-Agent string for the connection:

connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");

Once you do that, you will see the contents:

{"result":"ok","data":{"buy":174.03,"sell":160.64,"timestamp":"2017-12-24T04:21:00.326+00:00"}}

score 1 · Answer 2 · answered Dec 24 '17 at 05:09

The same achieved easily in Java 9 Http Client.

 try {
        HttpClient httpClient = HttpClient.newHttpClient(); //Create a HttpClient
        HttpRequest httpRequest = HttpRequest.newBuilder().uri(new URI("https://cws.hellogold.com/api/v2/spot_price.json")).GET().build();
        HttpResponse < String > httpResponse = httpClient.send(httpRequest, HttpResponse.BodyHandler.asString());

        System.out.println(httpResponse.body().toString());

    } catch (Exception e) {

        e.printStackTrace();
    }

Please Note while creating a java 9 project "requires jdk.incubator.httpclient;" Needs to be added in the module-info.java

Example:

module Http2JavaApiExamples {
    requires jdk.incubator.httpclient;
}

score 0 · Answer 3 · answered Dec 24 '17 at 04:27

A 403 errors means that website refused your connection. I looked around a little bit and found this: 403 Forbidden with Java but not web browser? It provides a working example that's easy to implement. All you have to do is specify your User-Agent property using the URLConnection#setRequestProperty method:

    URL urll = new URL("https://cws.hellogold.com/api/v2/spot_price.json"); 
    HttpURLConnection httpURLConnection = (HttpURLConnection) urll.openConnection();
    httpURLConnection.setRequestProperty("User-Agent", "Chrome");
    InputStream inputStream = httpURLConnection.getInputStream();
    BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));

Java parse url error

3 Answers3