Python - Requests pulling HTML instead of JSON

Question

I'm building a Python web scraper (personal use) and am running into some trouble retrieving a JSON file. I was able to find the request URL I need, but when I run my script (I'm using Requests) the URL returns HTML instead of the JSON shown in the Chrome Developer Tools console. Here's my current script:

import requests
import json

url = 'https://nytimes.wd5.myworkdayjobs.com/Video?clientRequestID=1f1a6071627946499b4b09fd0f668ef0'

r = requests.get(url)
print(r.text)

Completely new to Python, so any push in the right direction is greatly appreciated. Thanks!

`requests` can parse JSON for you. Take a look at what `r.json()` returns. — PM 2Ring, Feb 05 '18 at 20:53
@Shelby Also, having looked at the link, what made you expect it to return a JSON response? Is there a documented API? — roganjosh, Feb 05 '18 at 20:57
Possible duplicate of [HTTP requests and JSON parsing in Python](https://stackoverflow.com/questions/6386308/http-requests-and-json-parsing-in-python) — leo, Feb 05 '18 at 20:57
Hi, @roganjosh. The preview in the Developer Tools returns JSON. — Shelby, Feb 05 '18 at 21:01

deif · Accepted Answer · 2018-02-05T21:05:27.810

11

Looks like that website returns the response depending on the accept headers provided by the request. So try:

import requests
import json

url = 'https://nytimes.wd5.myworkdayjobs.com/Video?clientRequestID=1f1a6071627946499b4b09fd0f668ef0'

r = requests.get(url, headers={'accept': 'application/json'})
print(r.json())

You can have a look at the full api for further reference: http://docs.python-requests.org/en/latest/api/.

edited Feb 05 '18 at 21:05

answered Feb 05 '18 at 21:02

deif

1,349
1
14
19

Perfect! That did the trick. Thank you so, so much—much appreciated! – Shelby Feb 05 '18 at 21:04
1

sometimes `'User-Agent': 'PostmanRuntime/7.28.4'` in headers – lava Feb 15 '22 at 10:57

Python - Requests pulling HTML instead of JSON

1 Answers1