1

I'm attempting to write a web scraper here and the website is returning a 403 forbidden to my code even though it is an accessible webpage through a browser. My main question is: is this something that they set up on the website to discourage web scraping or am I doing something wrong?

import java.net.*;
import java.io.*;

public class Main {
    public static void main(String[] args) throws Exception {

        URL oracle = new URL("http://www.pcgs.com/prices/");
        BufferedReader in = new BufferedReader(
        new InputStreamReader(oracle.openStream()));

        String inputLine;
        while ((inputLine = in.readLine()) != null)
            System.out.println(inputLine);
        in.close();
    }
}

If I change the url to a website like http://www.google.com then it will return html. If the site is blocking it is there a way around that? Thanks for the help

Dommol
  • 191
  • 1
  • 17
  • 1
    Yes. You're doing something they are actively discouraging. Sure, you could probably use a tool like selenium to bypass it... but when/if they detect it they'll probably block your IP address. – Elliott Frisch May 15 '14 at 15:38
  • Thank you, I decided the better route would be to find a page that doesn't block my program. – Dommol May 15 '14 at 17:16

2 Answers2

2

Don't know much Java, but this simple Python code worked when I tried it without an error, saving the content as it appeared in my browser:

import requests                                                                            

r = requests.get('http://www.pcgs.com/prices/')    

with open('out.html', 'w') as f:
    f.write(r.content)

This sends a slightly unusual, non-browser user-agent.

So, if their site isn't likely blocking you on the basis of user-agent, maybe you've hit the site too quickly and they've blocked your IP address or rate limited you? If you're intending on scraping sites, you should be nice and limit the number of requests you make.

Another thing you can do before scraping is check for a site's robots.txt; like this one for Stack Overflow; that explicitly declares what the site's policies are towards automated scrapers. (In this case, the PCGS site doesn't appear to have one.)

Steven Maude
  • 856
  • 11
  • 21
1

The web server may contain a code to block not authorized user-agent.

I guess you can verify this by making sure your program will send a standard User-Agent value (i.e. corresponding to an existing web browser) and see if it makes any difference.

Datadecision team member
  • 317
  • 1
  • 6