1

I'm writing code that needs to reach out to this JSON file that's online, query it for data, and return the results of the search.

I tried the following code:

    adsb_data = urllib2.urlopen('http://website.com')

That will throw a urllib2.HTTPError: HTTP Error 403: Forbidden

So obviously I'm not going about it the best way. What would be the best way to open that JSON page, scrape it, and display results? I feel like there's a much more elegant and pythonic way to do this.

OutlawBandit
  • 67
  • 8
  • 1
    This is rather strange. I can reproduce the error with urllib. But it works fine using requests http://docs.python-requests.org – masterfloda Dec 19 '17 at 00:14
  • 1
    this work when I try it, might work for you too: [urllib2.HTTPError: HTTP Error 403: Forbidden](https://stackoverflow.com/a/13303773/1248974) – chickity china chinese chicken Dec 19 '17 at 00:15
  • HTTP 403 is very misleading. It should be any other 4xx code. Even 400 would be more appropriate... – masterfloda Dec 19 '17 at 00:18
  • 403 is appropriate if it's blocking the user-agent (using the semantics of: "your user agent is forbidden"). 400 may be ok (because technically the request is malformed from the API's point of view (bad User Agent)). – cowbert Dec 19 '17 at 01:38

2 Answers2

2

Looks like that particular server is blocking Python's User Agent. This should work for you:

url = "http://public-api.adsbexchange.com/VirtualRadar/AircraftList.json"
dummy_ua = "Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11" 
request = urllib2.Request(url, headers={'User-Agent' : dummy_ua}) 
urllib2.urlopen(request).read()
trinth
  • 5,919
  • 9
  • 40
  • 45
  • That's a really interesting solution. I hadn't even considered that it might be blocking the user agent. Since I wasn't in a browser it didn't even cross my mind. Thank you! – OutlawBandit Dec 19 '17 at 23:40
0

All good answers! Thank you everyone for input.

I fixed the issue by using requests instead of urllib2. Here is what my request looked like:

 adsb_data = http_request('http://public-api.adsbexchange.com/VirtualRadar/AircraftList.json', 'GET')

From there, I created a list and parsed out the data I wanted out of it.

OutlawBandit
  • 67
  • 8