0

I'm trying to parse this website with Jsoup: http://www.listofnewspapers.com/2014/04/site-map.html but I am getting this error:

For 'http://www.listofnewspapers.com/2014/04/site-map.html': HTTP error fetching URL"

I believe the error is occurring atconnect(URL).get().

I printed the status code for debugging purposes, and I got 200.

I tried with multiple other websites, and they connected fine.

Frits
  • 7,341
  • 10
  • 42
  • 60
MUZ
  • 1
  • 2

2 Answers2

1

It seems like user agents are filtered. Try setting up one:

Jsoup.connect(URL).userAgent("Mozilla").get()
Ilario Pierbattista
  • 3,175
  • 2
  • 31
  • 41
0

Use the Firefox userAgent:

.userAgent("Mozilla")

For example:

Jsoup.connect("https://stackoverflow.com/questions/44349297/http-error-fetching-url").userAgent("Mozilla").get();

Source: Can Jsoup simulate a button press?

João Rodrigues
  • 25
  • 1
  • 10