JSOUP Blocked: Returning HTTPStatusException

Asked Aug 29 '17 at 03:10

Active Aug 29 '17 at 03:49

Viewed 334 times

I keep getting HTTPStatusExceptions, whether it be 500/502/503, even 522. I'm guessing my IP has been blacklisted? What options do I have to ameliorate this?

I've noticed that while actually browsing the site, it takes forever to load, and that trying to view the source code actually times-out.

public Document getTPBDocument(String searchField) throws IOException {

Connection.Response response =  Jsoup.connect("https://thepiratebay.org/search/" + searchField + "/0/99/0").userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21")
            .referrer("http://www.google.com")
            .timeout(30000)
            .followRedirects(true)
            .header("Content-Type", "application/json;charset=UTF-8")
            .execute();
    return response.parse();

}

edited Aug 29 '17 at 03:49

asked Aug 29 '17 at 03:10

Ryan G

Can you share error trace also? – soorapadman Aug 29 '17 at 03:13
I added the trace. – Ryan G Aug 29 '17 at 03:23
Look like the way you are calling the link is wrong .Not sure without the code – soorapadman Aug 29 '17 at 03:46
Added the code. – Ryan G Aug 29 '17 at 03:49
Have you tried with userAgent `.userAgent("Mozilla")`? – soorapadman Aug 29 '17 at 03:52
look like this error https://stackoverflow.com/questions/36780047/java-jsoup-error-fetching-url – soorapadman Aug 29 '17 at 03:52
Trying it now, coming back with "java.net.SocketTimeoutException: Read timed out" – Ryan G Aug 29 '17 at 03:54
I tried using a different parsing library (Jaunt), and I'm getting the same issues. – Ryan G Aug 29 '17 at 04:14
I Just tried this its working ;`Document document = Jsoup.connect("https://thepiratebay.org/search/movies/0/99/0").userAgent("Mozilla").timeout(10*100000).get();` – soorapadman Aug 29 '17 at 04:18
let me know if it is not working – soorapadman Aug 29 '17 at 04:19
Yeah, still no luck. If I was blacklisted, how was it done, and how can I get past it? This is for a small personal project to learn more about webservices and scraping. I'm now getting an "IOException: Premature EOF" – Ryan G Aug 29 '17 at 04:24
I tried the above code which is working for me . May be You might have some restriction on your network for the site I have no clue . Im sure its not a code problem . – soorapadman Aug 29 '17 at 04:27
Darn, if only I could figure out what the restriction is. :-( – Ryan G Aug 29 '17 at 04:28
`EOF exception` could be because the server is closing the connection. – soorapadman Aug 29 '17 at 04:33
I wonder what's causing it to close the connection. Perhaps I need to change my IP. – Ryan G Aug 29 '17 at 04:35
If you have a non static IP address just reset your router and normally you'll receive a "new" one. Otherwise, did you try to set the timeout to 0 (which means forever)? / Another option would be to run this code on one of your friends' pc - to verify this is a network issue? – r3dst0rm Aug 29 '17 at 09:35
I'm going to try this after work, and I'll get back to you. – Ryan G Aug 29 '17 at 13:13

JSOUP Blocked: Returning HTTPStatusException

0 Answers0