403 status on GET call on API

Question

Please, help me out. I've just started developing and came up with a work related interresting idea to automate import duties calculation by getting the right duties through a API.

I'm trying to do a GET call from a API for import duties calculations. So I've watched some tutorials on YT and this is what I came up with:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.*;

public class Main {

    private static HttpURLConnection connection;

    public static void main(String[] args) {
        // Method 1: java.net.httpURLconnection
        BufferedReader reader;
        String line;
        StringBuffer responseContent = new StringBuffer();

        try {
            //URL url = new URL("https://jsonplaceholder.typicode.com/albums"); THIS ONE WORKS
            URL url =  new URL("https://www.tariffnumber.com/api/v1/cnSuggest?term=85"); HERE 403
            //connection.setRequestProperty("User-agent", "UTF-8");
            connection = (HttpURLConnection) url.openConnection();

            //request set up
            connection.setRequestMethod("GET");
            connection.setConnectTimeout(5000);
            connection.setReadTimeout(5000);

            int status = connection.getResponseCode();

            System.out.println(status);

            if(status > 299) {
                reader = new BufferedReader(new InputStreamReader(connection.getErrorStream()));
                while((line = reader.readLine()) != null) {
                    responseContent.append(line);
                }
                reader.close();
            } else {
                reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
                while((line = reader.readLine()) != null) {
                    responseContent.append(line);
                }
                reader.close();
            }

            System.out.println(responseContent.toString());

        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }   finally {
            connection.disconnect();
        }
    }

}

So my problem is that with the first URL (https://jsonplaceholder.typicode.com/albums) everything works fine. With the second URL (https://www.tariffnumber.com/api/v1/cnSuggest?term=85) I'm getting status 403. The weird thing is when I use the same URL in Postman and do a GET request, I get a 200 OK status. How is it possible that I get a 403 status in Java program and a 200 OK in Postman?

By the way, there is litteraly 0 documentation with this API:

https://www.tariffnumber.com/services/api

Thank you in advance.

EDIT: If I add

connection.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31");

right after setting the new URL, I'm getting a Exception in thread "main" java.lang.NullPointerException at Main.main(Main.java:53)

This is on the line of code

connection.disconnect();

This is weird because if I use the URL that does neither work. So adding the line of code is giving me a error later on.

EDIT2:

Fixed it with the help of your answers, the following piece of code should work:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.*;

public class Main {

    public static void main(String[] args) throws IOException {
        // Method 1: java.net.httpURLconnection

        BufferedReader reader;
        String line;
        StringBuffer responseContent = new StringBuffer();
        HttpURLConnection connection = null;

        try {
            //URL url = new URL("https://jsonplaceholder.typicode.com/albums");
            URL url =  new URL("https://www.tariffnumber.com/api/v1/cnSuggest?term=85");
            connection = (HttpURLConnection) url.openConnection();
            connection.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31");

            //request set up
            connection.setRequestMethod("GET");
            connection.setConnectTimeout(5000);
            connection.setReadTimeout(5000);

            int status = connection.getResponseCode();

            System.out.println(status);

            if(status > 299) {
                reader = new BufferedReader(new InputStreamReader(connection.getErrorStream()));
                while((line = reader.readLine()) != null) {
                    responseContent.append(line);
                }
                reader.close();
            } else {
                reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
                while((line = reader.readLine()) != null) {
                    responseContent.append(line);
                }
                reader.close();
            }

            System.out.println(responseContent.toString());

        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }   finally {
            connection.disconnect();
        }
    }

}

score 2 · Accepted Answer · answered Nov 23 '19 at 13:10

2

This API seems like a very badly implemented one: it doesn't seem to have any documentation, and although it's supposed to be free, it doesn't explain why it sends a 403 Forbidden status code.

I experimented and it seems it rejects the default User-Agent header sent by Java (Java/1.8.0_212 on my machine).

If you cheat and send the user agent of Chrome for example, it works fine:

connection.setRequestProperty("User-agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31");

We europeans pay taxes to provide such services. You should contact them so that they provide a decent documentation, and stop rejecting user agents like that.

answered Nov 23 '19 at 13:10

JB Nizet

678,734
91
1,224
1,255

Thank you for your answer and totally agree that they should add documentation. If I add the connection.setRequestProperty call where I've commented it out and run the program, I'm getting a Exception in thread "main" java.lang.NullPointerException at Main.main(Main.java:53) – KaiHoogenhoud Nov 23 '19 at 13:54
You can only set the request property on a connection once the connection has been created. Not before. You wouldn't have that kind of bug if you declared the variables only when you need them, when they're also initialized, rather than declaring them at the beginning of the method, or worse, as static variables. Delete that static variable, and use a local variable instead: `HttpURLConnection connection = (HttpURLConnection) url.openConnection();` – JB Nizet Nov 23 '19 at 13:58
see my editted code above. Right now it is working so thanks. – KaiHoogenhoud Nov 23 '19 at 14:22

score 0 · Answer 2 · answered Nov 23 '19 at 13:10

please refer [https://stackoverflow.com/a/13680048/11207493] and [Setting user agent of a java URLConnection

Below code should help.

connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");

403 status on GET call on API

2 Answers2